installing gems with jar dependencies like psych fails on recent ruby-maven/ruby-maven-libs

[jsvd]

I found that trying to install gems that require jars using modern ruby-maven and ruby-maven-libs wont work.
However JRuby will pull older versions by default:

/tmp/test# jruby -v
jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-Ubuntu-1ubuntu124.04 on 21.0.5+11-Ubuntu-1ubuntu124.04 +jit [aarch64-linux]
/tmp/test# gem list | grep maven
/tmp/test#

/tmp/test# cat Gemfile
source "https://rubygems.org"

gem "jar-dependencies"
gem "ruby-maven", "= 3.9.3"
gem "ruby-maven-libs", "= 3.9.9"

/tmp/test# bundle install --path=vendor/bundle
[...]
/tmp/test# find . -name maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/lib/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.9.3/lib/maven/ruby/maven.rb

/tmp/test# cat Gemfile
source "https://rubygems.org"

gem "jar-dependencies"
gem "ruby-maven", "= 3.9.3"
gem "ruby-maven-libs", "= 3.9.9"
gem "psych"

/tmp/test# bundle install --path=vendor/bundle
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Using jar-dependencies 0.4.1
Using bundler 2.3.26
Using ruby-maven-libs 3.9.9
Using ruby-maven 3.9.3
Fetching date 3.4.1 (java)
Installing date 3.4.1 (java)
Fetching psych 5.2.3 (java)
Installing psych 5.2.3 (java)
  jar dependencies for psych-5.2.3-java.gemspec . . .
Installing gem 'ruby-maven' . . .

using maven for the first time results in maven
downloading all its default plugin and can take time.
as those plugins get cached on disk and further execution
of maven is much faster then the first time.

2025-01-20T15:33:43.479Z [main] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
      org.snakeyaml:snakeyaml-engine:2.9:compile
Bundle complete! 4 Gemfile dependencies, 6 gems now installed.
Bundled gems are installed into `./vendor/bundle`

/tmp/test# find . -name maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/lib/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.9.3/lib/maven/ruby/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.3.13/lib/maven/ruby/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.3.9/lib/maven.rb

As we can see it works because older versions were still pulled even though I didn't ask for them:

However if I delete the older versions:

/tmp/test/vendor/bundle/jruby/3.1.0/gems# rm -rf psych-5.2.3-java/ ruby-maven-3.3.13 ruby-maven-libs-3.3.9/
/tmp/test/vendor/bundle/jruby/3.1.0/gems# cd /tmp/test/
/tmp/test# find . -name maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/lib/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.9.3/lib/maven/ruby/maven.rb

And try again, now we can see it fail:

/tmp/test# bundle install --path=vendor/bundle
[DEPRECATED] The `--path` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local path 'vendor/bundle'`, and stop using this flag
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Using jar-dependencies 0.4.1
Using bundler 2.3.26
Using date 3.4.1 (java)
Using ruby-maven-libs 3.9.9
Using ruby-maven 3.9.3
Installing psych 5.2.3 (java)
  jar dependencies for psych-5.2.3-java.gemspec . . .
      org.snakeyaml:snakeyaml-engine:2.9:compile

ERROR TEMPLATE:

Errno::ENOENT: No such file or directory - /root/.m2/repository/org/snakeyaml/snakeyaml-engine/2.9/snakeyaml-engine-2.9.jar -- module org.snakeyaml.engine.v2
  org/jruby/RubyIO.java:1278:in `sysopen'
  org/jruby/RubyFile.java:366:in `initialize'
  org/jruby/RubyClass.java:949:in `new'
  org/jruby/RubyIO.java:1195:in `open'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:504:in `copy_file'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:423:in `block in cp'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1577:in `block in fu_each_src_dest'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1593:in `fu_each_src_dest0'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1575:in `fu_each_src_dest'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:422:in `cp'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:92:in `vendor_file'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:151:in `block in vendor_jars'
  org/jruby/RubyArray.java:1981:in `each'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:150:in `vendor_jars'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:226:in `do_install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:170:in `vendor_jars'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/post_install_hook.rb:28:in `block in <main>'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/rubygems/installer.rb:381:in `block in run_post_install_hooks'
  org/jruby/RubyArray.java:1981:in `each'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/rubygems/installer.rb:380:in `run_post_install_hooks'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/rubygems_gem_installer.rb:43:in `install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/source/rubygems.rb:207:in `install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/gem_installer.rb:54:in `install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/parallel_installer.rb:186:in `do_install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:62:in `apply_func'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:57:in `block in process_queue'
  org/jruby/RubyKernel.java:1725:in `loop'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:54:in `process_queue'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:91:in `block in create_threads'

Environment

Bundler       2.3.26
  Platforms   ruby, universal-java-21
Ruby          3.1.4p0 (2024-11-04 revision 547c6b150eb2c15799349ba727d036cbb62a4069) [universal-java-21]
  Full Path   /tmp/jruby-9.4.9.0/bin/jruby
  Config Dir  /tmp/jruby-9.4.9.0/etc
RubyGems      3.3.26
  Gem Home    /tmp/test/vendor/bundle/jruby/3.1.0
  Gem Path    /tmp/test/vendor/bundle/jruby/3.1.0
  User Home   /root
  User Path   /root/.local/share/gem/jruby/3.1.0
  Bin Dir     /tmp/test/vendor/bundle/jruby/3.1.0/bin
OpenSSL
  Compiled    JRuby-OpenSSL 0.15.0
  Loaded      JRuby-OpenSSL 0.15.0
  Cert File   /usr/lib/jvm/java-21-openjdk-arm64/lib/security/cacerts
  Cert Dir    /etc/ssl/certs
Tools
  Git         not installed
  RVM         not installed
  rbenv       not installed
  chruby      not installed
Gem.ruby      /tmp/jruby-9.4.9.0/bin/jruby
bundle #!     /usr/bin/env jruby

Bundler Build Metadata

Built At          2022-11-17
Git SHA           23ec5b8501
Released Version  true

Bundler settings

path
  Set for your local app (/tmp/test/.bundle/config): "vendor/bundle"

Gemfile

Gemfile

source "https://rubygems.org"


gem "jar-dependencies"
gem "ruby-maven", "= 3.9.3"
gem "ruby-maven-libs", "= 3.9.9"
gem "psych"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    jar-dependencies (0.4.1)
    ruby-maven (3.9.3)
      ruby-maven-libs (~> 3.9.9)
    ruby-maven-libs (3.9.9)

PLATFORMS
  universal-java-21

DEPENDENCIES
  jar-dependencies
  ruby-maven (= 3.9.3)
  ruby-maven-libs (= 3.9.9)

BUNDLED WITH
   2.3.26

I believe the reason this fails is because the new mvn libs is outputting text the gems are not expecting. I got this from Logstash:

❯ MAVEN_HOME= ./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/maven-home/bin/mvn -q dependency:copy-dependencies dependency:list -Dmaven.repo.local=/Users/joaoduarte/.m2/repository -f /Users/joaoduarte/elastic/logstash/vendor/jruby/lib/ruby/stdlib/jars/gemspec_pom.rb -Djars.0=org.snakeyaml:snakeyaml-engine:jar:2.7 -Djars.specfile=/Users/joaoduarte/elastic/logstash/vendor/bundle/jruby/3.1.0/specifications/psych-5.2.2-java.gemspec -DoutputAbsoluteArtifactFilename=true -DincludeTypes=jar -DuseRepositoryLayout=true -DoutputDirectory=/Users/joaoduarte/.m2/repository -DoutputFile=/Users/joaoduarte/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.2.2-java/deps.lst && cat /Users/joaoduarte/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.2.2-java/deps.lst
artifact 0: {:type=>"jar", :group_id=>"org.snakeyaml", :artifact_id=>"snakeyaml-engine", :version=>"2.7"}
      org.snakeyaml:snakeyaml-engine:2.7:compile

The following files have been resolved:
   org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar -- module org.snakeyaml.engine.v2

mvn will output org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar -- module org.snakeyaml.engine.v2, which then ends up being treated as the dependency's full name, so then FIleUtils.copy breaks:

Errno::ENOENT: No such file or directory - /root/.m2/repository/org/snakeyaml/snakeyaml-engine/2.9/snakeyaml-engine-2.9.jar -- module org.snakeyaml.engine.v2
  org/jruby/RubyIO.java:1278:in `sysopen'
  org/jruby/RubyFile.java:366:in `initialize'
  org/jruby/RubyClass.java:949:in `new'
  org/jruby/RubyIO.java:1195:in `open'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'

Let me know if I should move this to one of the jruby maven gem repositories instead.

[jsvd]

I have added a patch in Logstash to strip the output of maven https://github.com/elastic/logstash/pull/16919/files#diff-9e78c34e57f2d0ebb1b85aab010e6d6f55101da6da4f1c324f2246dcf334a366R137-R158

Example output:

DEBUG: load_from_maven called with arguments: "/private/tmp/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.2.2-java/deps.lst"
changed from ""   org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar\e[36m -- module org.snakeyaml.engine.v2\e[m\n"" to ""org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar""
DEBUG: load_from_maven returned: [#<Jars::Installer::Dependency:0x1b2e2589 @gav="org.snakeyaml:snakeyaml-engine:2.7", @system=false, @type=:jar, @coord="org.snakeyaml:snakeyaml-engine:jar:2.7", @file="/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar", @scope=:runtime, @path="org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar">]

[headius]

This will be fixed momentarily when I release an update to jar-dependencies.

[headius]

Fixed by jar-dependencies 0.5.2.

[jsvd]

Thanks, I'm still hitting some issues with 9.4.10.0, so we may wait until a new release ships with jar-dependencies 0.5.2.

I'm not sure why:

FROM jruby:9.4.10.0
WORKDIR /tmp
RUN gem install 'jar-dependencies' -v '0.5.2'
RUN gem install 'psych' -v '5.2.2'

Works correctly, but:

FROM openjdk:21
WORKDIR /tmp
RUN curl -s https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.10.0/jruby-dist-9.4.10.0-bin.tar.gz | tar -zxf -
ENV PATH="/tmp/jruby-9.4.10.0/bin:$PATH"
RUN gem install 'jar-dependencies' -v '0.5.2'
RUN gem install 'psych' -v '5.2.2'

Will fail with the same error:

jruby/jruby#8 6.730 using maven for the first time results in maven
jruby/jruby#8 6.730 downloading all its default plugin and can take time.
jruby/jruby#8 6.730 as those plugins get cached on disk and further execution
jruby/jruby#8 6.730 of maven is much faster then the first time.
jruby/jruby#8 6.730
jruby/jruby#8 11.73       org.snakeyaml:snakeyaml-engine:2.7:compile
jruby/jruby#8 14.35 ERROR:  While executing gem ... (Errno::ENOENT)
jruby/jruby#8 14.35     No such file or directory - /root/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar.[36m -- module org.snakeyaml.engine.v2.[m
jruby/jruby#8 14.35        org/jruby/RubyIO.java:1279:in `sysopen'
jruby/jruby#8 14.35        org/jruby/RubyFile.java:371:in `initialize'
jruby/jruby#8 14.35        org/jruby/RubyClass.java:951:in `new'
jruby/jruby#8 14.35        org/jruby/RubyIO.java:1196:in `open'
jruby/jruby#8 14.35        /tmp/jruby-9.4.10.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'

[headius]

Wait a bit longer... there's some issues with the 0.5.2 release I'm working through today.

[jsvd]

@headius just fyi the extra text in the output of maven deps is still causing an issue even with jruby 9.4.11.0 and jar-dependencies 0.5.3. Given the Dockerfile:

FROM openjdk:21
WORKDIR /tmp
RUN curl -s https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.11.0/jruby-dist-9.4.11.0-bin.tar.gz | tar -zxf -
ENV PATH="/tmp/jruby-9.4.11.0/bin:$PATH"
RUN mkdir /tmp/my_gem
RUN echo -e "Gem::Specification.new do |spec|\n" \
    "  spec.name          = 'my_gem'\n" \
    "  spec.version       = '0.1.0'\n" \
    "  spec.summary       = 'A simple gem that downloads a JAR'\n" \
    "  spec.authors       = ['Your Name']\n" \
    "  spec.email         = ['your@email.com']\n" \
    "  spec.files         = Dir['lib/**/*']\n" \
    "  spec.require_paths = ['lib']\n" \
    "  spec.add_dependency 'jar-dependencies'\n" \
    "  spec.requirements = 'jar com.google.protobuf:protobuf-java, 4.29.3'\n" \
    "  spec.platform = 'java'\n" \
    "end" > my_gem.gemspec
RUN cat my_gem.gemspec
RUN gem build -q my_gem.gemspec
RUN gem install my_gem-0.1.0-java.gem -N

errors with:

jruby/jruby#12 [8/8] RUN gem install my_gem-0.1.0-java.gem -N
jruby/jruby#12 3.789   jar dependencies for my_gem-0.1.0-java.gemspec . . .
jruby/jruby#12 3.790 Installing gem 'ruby-maven' . . .
jruby/jruby#12 4.934 Successfully installed ruby-maven-libs-3.9.9
jruby/jruby#12 5.021 Successfully installed ruby-maven-3.9.3
jruby/jruby#12 5.022
jruby/jruby#12 5.022 using maven for the first time results in maven
jruby/jruby#12 5.022 downloading all its default plugin and can take time.
jruby/jruby#12 5.022 as those plugins get cached on disk and further execution
jruby/jruby#12 5.022 of maven is much faster then the first time.
jruby/jruby#12 5.022
jruby/jruby#12 10.51       com.google.protobuf:protobuf-java:4.29.3:compile
jruby/jruby#12 14.23 ERROR:  While executing gem ... (Errno::ENOENT)
jruby/jruby#12 14.23     No such file or directory - /root/.m2/repository/com/google/protobuf/protobuf-java/4.29.3/protobuf-java-4.29.3.jar.[36m -- module com.google.protobuf.[0;1m [auto].[m
jruby/jruby#12 14.23       org/jruby/RubyIO.java:1279:in `sysopen'
jruby/jruby#12 14.23       org/jruby/RubyFile.java:371:in `initialize'
jruby/jruby#12 14.23       org/jruby/RubyClass.java:951:in `new'
jruby/jruby#12 14.23       org/jruby/RubyIO.java:1196:in `open'
jruby/jruby#12 14.23       /tmp/jruby-9.4.11.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'

[headius]

Ok I will try to reproduce and we will get this fixed. I'm going to move this to the jar-dependencies repo. At least we can upgrade it now!