Changes to use externally installed GDAL library #42
Conversation
|
I made changes to the "nbproject" files that worked for me. But I am new to Netbeans, so those changes might not be quite right. There is also a file for IntelliJ (.idea/libraries/gdal.xml) that will need to change, and I have no idea how to do that. The build check will fail, of course, since it most likely doesn't have the GDAL packages installed. |
|
@gbburkhardt, I have a made a couple of changes that would allow you to build the project with Gradle in NetBeans. You just need to install the Gradle plugin in NetBeans. I can push these changes to your branch if you want to. Gradle allows us to fetch the GDAL jar from the maven repositories. So, these changes fits nicely with the work you did in this commit. |
|
Sure, please merge in your Gradle changes. That's the direction that the team is going, right? Away from Ant builds and toward Gradle builds? Will both be maintained? |
|
I notice that you've removed the 'nbproject' directory. Should the .idea directory be removed as well, or will someone maintain it? |
|
I'm not too familiar with IntelliJ, but I think the |
|
There is a problem with using 'gdal.jar' from Maven. One has to make sure that there's a matching set of native libraries on the system. In particular, native libraries for GDAL 2.2.3 or earlier won't work with GDAL 2.2.4 or later. Although the Java API probably doesn't change much over time, just as the C++ API is pretty well set, it could. So it's better to get 'gdal.jar' that comes with the set of native libraries. |
|
@gbburkhardt, to compile the project with Gradle, we need to specify the location of a GDAL jar, or else the compilation will fail. It could be either a file on disk (such as the old GDAL jar file that was located in the WorldWind directory), or an external Maven dependency. I thought the easiest would be an external Maven dependency as this would just work out of the box. The different versions are available at: https://mvnrepository.com/artifact/org.gdal/gdal. I just used the latest version (2.4.0) to compile against. This jar is only used during compilation and not loaded at runtime, hence the |
|
@gbburkhardt, I looked a bit at what you did with the ant-script with respect to locating the GDAL jar. I'll try to see if I can modify the Gradle build file to work in a similar way. It is probably best to build against the GDAL that is located on the user's system. It would have been nice to make use of the GDAL jar on Maven but, like you mentioned, there could be API differences etc. that could cause problems. I'll see what I can figure out. |
|
OK, I got the Travis builds to mostly succeed. The only error that is still thrown is for OpenJDK11 and OpenJDK-EA and looks like it is due to a certificate issue on Oracle's side (see: https://stackoverflow.com/questions/55878548/travis-ci-couldnt-install-openjdk11). |
|
@gbburkhardt, the only issue that I can think of with regards to building against a GDAL jar on the user's system is if the user just wants to build WorldWind without making use of any GDAL functionality. Then, if we go the GDAL system jar route, the user needs to have a working GDAL installation on their system to be able to just build WorldWind. Whereas, if we pull in an external GDAL Maven dependency during the compilation phase, the user can still compile WorldWind without ever having to worry about GDAL. |
|
There's a lot of merit if the project builds easily without errors at the start, so perhaps pulling a copy of gdal.jar file from Maven isn't so bad. If an application isn't set up properly and tries to use GDAL, I think that the error messages are helpful enough for someone to quickly identify the issue. Then hopefully they'll read the GDAL_README.txt file. Probably I should put something in README.md, too. |
|
@gbburkhardt, I agree. I think we should strive to make WorldWind as easy as possible to build. If the user wants to make use of GDAL at runtime, then they need to know about installing GDAL and configuring WorldWind to make use of the system-based GDAL as outlined in |
|
@gbburkhardt, I tried to get the GDAL stuff running on my PC. But I'm having some issues. The script I'm using looks like: However, I get this error when running the ImportImagery example application: I'm running Ubuntu 18.04 Bionic. Is there anything I'm doing wrong to get this error. I see from the output that it complains that I haven't set the |
|
@gbburkhardt, I made a couple of fixes.
The rest was cleanups in terms of unused imports and some formatting fixes so that the code fits into the rest of the code-base. It looks like it starts up now and manages to find the GDAL data and plugin directories. This is how it looks after startup:
The stack-trace looks like this: Which shows that it managed to successfully detect the GDAL data and plugin directories. However, the image that is imported looks different compared to when it is imported with the old GDAL code (before this commit):
With this stack-trace: The new changes causes the black border in that image for some reason. Do you know what the reason for this could be? |
|
I installed the Groovy/Gradle plugin in Netbeans 11, and the build worked for me. I didn't notice the unit tests running, but presumably there's another option to do that. |
|
I was able to get the image to display properly by commenting out one line of code in GDALDataRaster.java:981 I'll need to do some research to figure out why. It makes a bit of sense - GDALUtils.ALPHA_MASK is 0xFFFFFFFFL (black). There must have been a change in the GDAL API. By the way, if you turn off the Crater Lake imagery, there's a blue background with a similar outline. But that's in the Landsat data. One can see that with the WorldWindow application:
|
|
@gbburkhardt, I think I need to remove some of the general Gradle and Travis changes I made from this branch and add it as a separate pull-request. I don't want to pollute this pull-request with some of the changes I made. I'm going to have to force-push this branch again if that is OK with you. You just need to then fetch the branch again. I think it it better to keep certain commits separate from one another to be able to properly keep track of things. |
|
@gbburkhardt, OK, I made the changes and removed the Gradle and Travis fluff that should not have been in this pull-request to begin with. I added them to pull-request #44. The commits in this branch now mostly have to do with getting GDAL to work. I think it is now easier to review the changes. |
|
Regarding the display of the Crater Lake surface image, this bit of code is a real conundrum: But it doesn't make sense to me that one would alter the values in the image at all. It looks to me as if that fill command is setting the entire alpha band to 255, or 100%, for the entire image. The next avenue for investigation would be to pull the GDAL 1.7.2 code, to see what 'fill' used to do for alpha bands. |
|
Sure love all those bands, rainbow for water depth and deep blue for the water surface. :~} |
|
I've spent the last few evenings on this, but haven't cracked it yet. Curiously, deleting the fill of the alpha band with 255 doesn't affect the display when GDAL 1.7.2 is used. I do notice that there is some special handling for alpha bands for GDAL rasters, but I don't quite understand the motivation. See GDALUtils.extractImageMask and GDALUtils.applyImageMask, and GDALDataRaster.createMaskDataset. But those methods don't seem to apply in this case, because of the 8 bit color data. Or perhaps they should, and it wasn't realized that the mask data values would be truncated to 8 bits due to ReprojectImage, so the applyImageMask never changes the alphaband... Anyway, I'm working on it and the resolution will take a while. |
|
Glenn, |
|
The problem with the black boundary around the crater lake image is indeed due to line 981 in GDALDataRaster.java. Please see this discussion. I don't know why that line of code was added, but I now believe that it is wrong. Deleting should have no side effect. It's in the method 'createCompatibleDataset', which is only called in one place, GDALDataRaster.doGetSubRaster(), to create the 'destDS'. The calls to 'ReprojectImage' look like they are supposed to copy the source dataset into the destination dataset with an appropriate transformation. But the alpha mask in the source should be retained. Given what Even Rouault says, setting all pixels in the alpha mask in 'destDS' to 255 will discard the pixels with alpha=0 in the source, which is exactly what we're seeing. I dumped the destination alpha mask for 'destDS' after the ReprojectImage calls using both the 1.7.2 and 2.4.0 GDAL versions. With 1.7.2, the 'destDS' was the same as the source alpha mask regardless of whether the 'destDS' alpha mask pixels were set to 0 or 255. But with 2.4.0, when set initially to 255, the 'destDS' alpha mask pixels were all 255. So I believe that the correct solution is to delete the fill, and that there shouldn't be side effects. |
|
@gbburkhardt, thanks. Everything looks good to me. I tested it on my side and it is working perfectly now. I removed my Gradle changed again from your @emxsys, I think we can merge this branch into |
|
I just noticed that the link I gave for GISinternals, which is for the version compiled for VS 2017, doesn't include the latest version of the ECW plugin. So I suggest pulling the files from here: GISinternals, and using these files: gdal-204-1900-x64-core.msi Note that some of the command line utilities in the GDAL directory are Python scripts, so if you want to use those, also use one of the GDAL-2.4.0.win-amd64-pyX.X.msi for the version of Python you're using. |
|
@wcmatthysen, you stated "We do not package the DLLs inside a jar anymore". I'm wondering who "we" is and why DLLs aren't packaged inside a jar anymore? Is this due to Java changes? Our application currently is distributed via Web Start. So the ideal situation would be for me to package everything needed for GDAL in jars just like it was being done in previous versions of WWJ. I understand the reasons for the WWJ-CE project moving away from this, but that approach would seem to be best for us... unless there's a reason why this isn't a good approach. Again, the primary reason for this is that we distribute software built on top of WWJ to clients via Web Start and don't want our clients to have to install/configure additional 3rd party software. Is it possible for me to package the required files for GDAL 2.4 into one or more jar files? I have the gdal.jar file from GISInternals. And I've created a gdaldata.jar file from the contents of the gdal-data folder. But I'm not sure how to recreate the gdal-natives-windows-amd64.jar file. The version that was in WWJ contains gdalalljni64.dll and gdalminjni64.dll. I find gdallalljni.dll in the download from GISInternals, but it is much smaller that the DLL in the old GDAL-natives. And I cannot find gdalminjni.dll. Bottom line is that I'd really like to package everything in jar file(s) if possible. Any thoughts and directions are very much appreciated! |
|
By the way a small edit to the GDAL_README would be good to update the URL at the end that is supposed to point to instructions regarding the use of WW GDAL libraries with a Java Web Start application. The current URL is https://goworldwind.org/getting-started/ (which is broken) It seems the URL should be https://worldwind.arc.nasa.gov/java/get-started/ |
|
@robmilton, when I refer to "we" I'm referring to the discussion we intially had regarding packaging GDAL ourselves. You can see the discussion in issue #17. It is not just about packaging the native libraries in a jar. If you look at the discussion you'll see that in the upstream repository of WorldWind they built their own custom monolithic GDAL DLL that pulls in everything that is required. This is how WorldWind has always pulled in GDAL functionality in the past. This is quite a cumbersome process to do for every release of GDAL (just ask @gbburkhardt). That is why we decided that the best approach moving forward would be to rely on the user to install GDAL on their system and we just hook into this GDAL. With regards to your question about rebuilding a native jar (such as If you don't mind using the old version of GDAL you can try this: I'm not sure if this will work but you can still try to use the upstream |
|
As the 'old' WorldWind project manager for NASA, now retired, it's great to see this fresh air being applied to the ongoing development. Seems like some pretty good wind for sailing ahead! Much appreciated!!! |
|
@robmilton, sorry. I see now with the webstart stuff we need to provide a jar with the native libraries. You can disregard what I said with regards to building a For the webstart case the native DLLs will still need to be loaded from a jar as far as I understand (my experience with webstart is limited). I think you can still manually create the @emxsys, @gbburkhardt, I'm not too familiar with the webstart stuff. Is this correct? |
|
@wcmatthysen, I've been reading through the issue/discussion you linked and am getting a better picture of the challenges. Still it seems for distributed applications that the idea of requiring the end user to install additional 3rd party software is not the most desirable... certainly not from our perspective anyway. Thinking about these changes is complicated even more by the Java 9+ changes taking place. And I know that one of those changes is that Web Start doesn't exist beyond Java 8. We've been working on a replacement for Web Start, but we're not ready to make that change yet. So I do need to come up with an approach that makes this seamless for our clients. Whatever I do needs to be done in our code/package... and for now that means being deployed by Web Start... so that our clients don't have to install/configure any additional software. You mentioned copying "the GDAL system DLLs" to a directory and creating the natives jar file. Can you help me identify exactly which DLLs/files should be included? Also I downloaded the "release-1900-x64-gdal-2-4-0-mapserver-7-2-2.zip" file from GISInternals hoping it would contain all I needed without having to install the 3 MSIs listed by @gbburkhardt. Do you happen to know if everything I need is contained in that ZIP or will I need to install the other the packages to get everything I need. Thank you for helping me through this issue! |
|
Something just dawned on me as I'm continuing to read back through this discussion... If we're not having any problems with the version of GDAL that was previously packaged with WWJ (I believe that's version 1.7.2 according to the NASA WWJ GDAL_README) and we don't have a need for the additional features/etc. in the new GDAL... is there anything preventing us from continuing to use and deliver the GDAL jars that were packaged with NASA WWJ as we have been? I guess I'm not clear on if there have been any changes to the WWJ-CE code that would prevent this from working? This just hit me as a possibility, so I haven't tried it yet. But it's something I can set up and try relatively easily. |
|
@robmilton, with regards to your first message: Yes, I haven't been aware of the webstart challenges as I'm not using WorldWind in a webstart environment. In fact, I have very limited experience with webstart. However, I checked and it seems that with webstart the native libraries needs to be packaged in a jar for it to work. Our approach of requiring the user to install 3rd party software is not the most desired solution. However, given the circumstances, we also do not have the capacity currently to maintain a monolithic GDAL DLL. If we want to go this route we need someone to volunteer to do this and to provide us with regular builds for all of the different platforms. On the other hand, if you want to build your own jar that includes the required DLLs you can start with With regards to your second message. That is exactly what I suggested one of my replies:
However, instead of unpacking the |
|
@wcmatthysen, I saw that suggestion but had disregarded it since it involved extracting all the DLLs and placing them in the root of my application directory... since I wasn't sure how to tackle that with Web Start. But I've compiled the latest WWJ-CE code and plan to test with our existing Web Start set up tomorrow. Thanks again for the help with this! |
It looks like that information might be lost forever, due to the shutdown of NASA's project. I don't see any GDAL specific instructions for Web start on https://worldwind.arc.nasa.gov/java/get-started. |
Of course your clients shouldn't have to install/configure additional software with your application. You should include everything needed in your release. I myself don't have the time to set up build scripts to pull in all the code needed to build a complete edition of GDAL. I spent about a week trying, and decided that a better approach was to rely on the efforts of others, who were already providing binary releases of GDAL. Folks will want to use the latest version of GDAL, and keeping up with the release and the dozens of different sources for the software for different sources would take a major effort. One would want to build a system that pulled in the code from the various sources and built it automatically. Given that good binary distributions are currently available, it makes more sense for a small group like ours to concentrate on the WorldWind core software. I don't have any experience with Web Start. But I don't think it would be too complicated to put together a script that built a jar file with the native libraries that could be used for Web Start applications. Perhaps you could do that, and contribute it to this effort. |
Hey Glenn, that link does contain a little information about using GDAL with Web Start. If you scroll to the bottom of the page it shows how to include an "extension" reference in your project's primary JNLP file. And you can still follow the URL shown which will download an example gdal.jnlp file. This file is actually a good example of how to include the GDAL jars in a WWJ application using Web Start. |
I'll definitely be working to figure out a solution of some sort. Whatever I figure out I'll be glad to share. As I mentioned elsewhere I am very backlogged with development tasks at work, but I definitely want to find time and ways to contribute to the effort. |
|
So compiling the latest WWJ-CE project changes and attempting to use with the previous GDAL jars did not work. When I attempt to load a geo-referenced image I see the message I've also experimented most of the day building jar files (1, 2, 3 jars) with every combination of contents and differing structures I can think of. However, every time I run the application and attempt to load a geo-referenced image I get the following error: Will continue to work on this. But any additional thoughts on how to either use the previous GDAL jars or create new GDAL jar(s) would be greatly appreciated. |
@wcmatthysen, can you point me to the code that was changed that performed the DLL unpacking from a jar? I'm wondering if I could add that back (into my branch at least) so that I can just add the GDAL folder structure back into "lib-externals" and use the jars from WWJ v2.1.0 for now? In the future when we get away from Web Start I can work on copying the various GDAL resources under the install location for our application and get the new GDAL approach and version working at that point. |
|
Should I enter an issue to look into this change and the impact on Web Start applications? I understand the reasons for the change, but the impact on those of us with applications distributed using Web Start is pretty huge. It forces us to have to figure out a solution to a problem that didn't exist prior to this change. |
I see it now. I'll get the README file updated, and perhaps copy the info on Webstart to the Wiki pages. The gdal.jnlp that's referenced there is the same as is in the 'webstart' subdirectory in the WorldWindEarth repository. |
The NASA WorldWind release didn't package the GDAL native libraries in jar files. The DLL's/.so's were in lib-external. So how did you get your webstart application working with those? The JOGL native libraries were (and still are) packaged in jar files.
One wouldn't expect the new code to work with the previous GDAL jar files. But do try this (per @wcmatthysen 's post of a few days ago:
Make sure that the new 'gdal.jar' file is in the application's class path. Change to the application launch directory. Invoke 'java' on the jar file or start the .exe file from there. I expect it will work. There are all sorts of ways to package this for an end user. I attempted to explain the basic principles in the GDAL_README.txt file. How you starting your application? What command would you issue from a command line? If you have a jar file that you're passing to 'java', another alternative is to
Refer to the file WorldWindJava\src\gov\nasa\worldwind\util\gdal\GDALUtils.java to see how it searches for the various components. I have toyed with the idea of setting the PATH environment and java.library.path programmatically, and putting the location of the files in worldwind.xml, but Java doesn't seem to be amenable to that approach. |
|
@gbburkhardt, thank you for the detailed response. The problem is that we're using Web Start to distribute/start the application. I don't know of a way to follow the steps you've described since AFAIK everything has to be in jar files when launched with Web Start. If I was just packaging up some jars with a bunch of files and installing them in a specified location then I could perform the steps you've described... but that's not the case. I need a solution that works with Web Start (which is the way we've been distributing/launching our WWJ-based app for over 7 years). |
So how did you package the GDAL DLL's? Those didn't come in jar files as supplied by NASA. |
|
In a post earlier today I mentioned:
I've experimented with packaging the files from the GDAL installation in numerous ways. No matter what I did I kept getting the error message above. And it appears that the JNLPClassLoader is not able to find the librar(ies). Since I don't really know which code is responsible for loading which items with Web Start I'm not sure how to package the elements. I tried every combination of every suggestion mentioned in this conversation... and quite a few more. |
|
The instructions I described were for standalone applications, not for web start. I have no experience with that technology at all. But you must have staff who put it together originally. What did they do with the DLLs in the lib-external/gdal that are in the NASA release? |
|
I am the staff that put it together originally. I didn't do anything with the DLLs that were in the lib-external/gdal folder. The NASA team (or someone) packaged jar files that were accessed via the GDAL.jnlp file mentioned earlier. So I followed the URL(s) in that JNLP file and downloaded every jar file referenced there. I then modified the manifest files and signed the jar files so that we could use them with our application. So there was nothing required on our end except the steps described above and then configuring our application's JNLP file. EDIT: That's why this is so difficult. This all "just worked" up until the change to use an externally installed GDAL library. And as you guys have mentioned this does not appear to be something that is easily remedied... and with the list of other pressing issues that we have to deal with this BREAKING CHANGE is a real pain. EDIT 2: To tell you the truth I wish this change would be rolled back until it can be accomplished without breaking WW for those of us using Web Start (since WW has worked with Web Start for many years). |
|
I used 'wget' to pull the worldwind.arc.nasa.gov site, but could find no jar files. There's a github repository for WebWorldWind, but again, no jar files. There is a site with several blubs about NASA WebWorldWind, which had a page referencing NASA's Artifactory site for web apps, no jar files. Sorry, out of time for tonight. |
|
@gbburkhardt, I'm assuming you're looking for the GDAL jar files I was describing that must have been put together by the NASA team? You can download NASA's gdal.jnlp file here. This is the location from which we downloaded the GDAL jar files years ago. So tonight I downloaded the gdal.jnlp file and then the jar files referenced inside it so that we'd have the originals from NASA. A few of the "nativelib" files referenced in the JNLP file aren't there, but I was able to download the following 7 files (which are the same ones we've been using for the past 6 or 7 years): I've zipped these 7 jar files and attached them to this post... There must be a way to package the new GDAL content in one or more jar files so that we can continue to launch WWJ both from the desktop and via Web Start. EDIT: I'm assuming this would work just like the jogl-all.jar along with the 5 jogl-all-natives jars and the gluegen-rt.jar along with the 5 gluegen-rt-natives jars. |
|
@robmilton, nothing stops you from still using the upstream WorldWind if you are pressed for time. We haven't made that many changes that it warrants an upgrade if you are looking for new features. We just upgraded JOGL, switched to the external version of GDAL and fixed a couple of small issues here and there. You can do the JOGL upgrade easily if you look at: NASAWorldWind#1. I'd suggest that you fall back on the upstream version until we actually have a 2.2.0 release ready. Also, can we move this discussion into a new separate issue. You continue to make comments on closed pull-requests. I suggest you open a new issue where everybody can easily locate it and participate in the discussion. When people switch to the pull-request or issues tab on GitHub they don't see the closed ones. |
|
Don't worry, I opened an issue for this. We should continue the discussion at issue #50. |
|
@wcmatthysen, thank you for creating the issue. I may end up having to switch back to WWJ v2.1.0, but I'd rather not. Our ultimate goal is to get our application which is built on top of WWJ working with the latest Java release. With the biggest challenge I believe being the WWJ part. So I'm very excited about the WWJ-CE project in hopes of getting there. I'm not particularly interested/needing new features at this time (although it's great to have them and to get upgrades!). But my fear of not using WWJ-CE is getting too far behind the changes. For instance in addition to the JOGL and GDAL changes the MercatorTiledImageLayer was recently changed... which is great!. But since we had extended that class it took a while to adjust our code to the changes. I'm sorry to present these challenges to the positive movement here. I'm the only full-time developer at my company, and trying to modify our code to comply with all the Java changes has become a huge issue. And of course new feature requests for our product are piling up (which also is great!), but it seems I keep running up against changes from every direction that pull the rug out from under me. Thanks for all you guys are doing! I'll use Issue #50 for further communications on this topic. |
Description of the Change
Modify GDALUtils to make use of GDAL libraries installed externally from WorldWind. Old version of GDAL removed from repository. Instructions included for making use of external GDAL libraries for Windows and Linux. Someone with access to Apple products will need to update appropriately.
Why Should This Be In Core?
The current GDAL 1.7.2 version was released in April 2010. Numerous improvements in the library have been made since then. Building GDAL is non-trivial. We're better off relying on existing binary distributions of GDAL going forward in order allow WorldWind to make use of current GDAL releases.
Benefits
Make current technology available to users.
Potential Drawbacks
WorldWind is no longer self contained. GDAL libraries will need to installed separately.
Applicable Issues
Users will need to configure WorldWind property files to use the installed locations of GDAL. The property files included work for the GISInternals binary release for Windows and for Ubuntu Bionic 18.04.