Add GPULlama3.java as model provider to run on GPUs #1793
Merged
+1,710
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Nice!! |
|
How would you like to to move this forward? |
I was expecting to first merge the PR and backport the latest state here. Since that might take some time, I can start preparing the backport in parallel and make the PR final, so you can test it. |
|
No rush on my end, completely up to you on how you want to proceed |
|
By following the steps in README.md Quarkus boot that does not load tornado-runtime properly: ❯ source ../../../TornadoVM/setvars.sh
❯ tornado --devices
WARNING: Using incubator modules: jdk.incubator.vector
Number of Tornado drivers: 1
Driver: OpenCL
Total number of OpenCL devices : 1
Tornado device=0:0 (DEFAULT)
OPENCL -- [Apple] -- Apple M4 Pro
Global Memory Size: 16.0 GB
Local Memory Size: 32.0 KB
Workgroup Dimensions: 3
Total Number of Block Threads: [256]
Max WorkGroup Configuration: [256, 256, 256]
Device OpenCL C version: OpenCL C 1.2
❯ mvn quarkus:dev -Djvm.args="@/Users/orion/TornadoVM/tornado-argfile"
[INFO] Scanning for projects...
[INFO]
[INFO] --< io.quarkiverse.langchain4j:quarkus-langchain4j-integration-test-gpu-llama3 >--
[INFO] Building Quarkus LangChain4j - Integration Tests - GPULlama3 999-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- quarkus:3.18.0:dev (default-cli) @ quarkus-langchain4j-integration-test-gpu-llama3 ---
[INFO] Invoking enforcer:3.5.0:enforce (enforce-java-version) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Rule 0: org.apache.maven.enforcer.rules.BannedRepositories passed
[INFO] Rule 1: org.apache.maven.enforcer.rules.version.RequireJavaVersion passed
[INFO] Invoking enforcer:3.5.0:enforce (enforce-maven-version) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion passed
[INFO] Invoking sundr:0.200.4:generate-bom (default) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Invoking buildnumber:3.2.1:create (get-scm-revision) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Executing: /bin/sh -c cd '/Users/orion/quarkus-langchain4j/integration-tests/gpullama3' && 'git' 'log' '-1' '--no-merges' '--format=format:%H %aI %aE %aN'
[INFO] Working directory: /Users/orion/quarkus-langchain4j/integration-tests/gpullama3
[INFO] Storing buildNumber: 3b53f8b6094e48dde6cc554540cd2d6aa1242f0f at timestamp: 1761131170758
[INFO] Executing: /bin/sh -c cd '/Users/orion/quarkus-langchain4j/integration-tests/gpullama3' && 'git' 'symbolic-ref' 'HEAD'
[INFO] Working directory: /Users/orion/quarkus-langchain4j/integration-tests/gpullama3
[INFO] Storing scmBranch: gpu-llama3-integration
[INFO] Invoking formatter:2.26.0:format (format-sources) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Processed 1 files in 167ms (Formatted: 0, Skipped: 1, Unchanged: 0, Failed: 0, Readonly: 0)
[INFO] Invoking impsort:1.12.0:sort (sort-imports) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Processed 1 files in 00:00.005 (Already Sorted: 1, Needed Sorting: 0)
[INFO] Invoking resources:3.3.1:resources (default-resources) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Copying 1 resource from src/main/resources to target/classes
[INFO] Invoking compiler:3.14.0:compile (default-compile) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] Nothing to compile - all classes are up to date.
[INFO] Invoking resources:3.3.1:testResources (default-testResources) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] skip non existing resourceDirectory /Users/orion/quarkus-langchain4j/integration-tests/gpullama3/src/test/resources
[INFO] Invoking compiler:3.14.0:testCompile (default-testCompile) @ quarkus-langchain4j-integration-test-gpu-llama3
[INFO] No sources to compile
[WARNING] [io.quarkus.bootstrap.devmode.DependenciesFilter] Live reload was disabled for the following project artifacts:
- io.quarkiverse.langchain4j:quarkus-langchain4j-gpu-llama3:999-SNAPSHOT
- io.quarkiverse.langchain4j:quarkus-langchain4j-core:999-SNAPSHOT
- io.quarkiverse.langchain4j:quarkus-langchain4j-core-runtime-spi:999-SNAPSHOT
- io.quarkiverse.langchain4j:quarkus-langchain4j-core-deployment:999-SNAPSHOT
- io.quarkiverse.langchain4j:quarkus-langchain4j-gpu-llama3-deployment:999-SNAPSHOT
The artifacts above appear to be either dependencies of non-reloadable application dependencies or Quarkus extensions
[WARNING] [io.quarkus.deployment.dev.DevModeCommandLineBuilder] org.graalvm.sdk:word::jar classpath entry /Users/orion/.m2/repository/org/graalvm/sdk/word/23.1.2/word-23.1.2.jar was overriden with /Users/orion/.m2/repository/org/graalvm/sdk/word/23.1.0/word-23.1.0.jar
[WARNING] [io.quarkus.deployment.dev.DevModeCommandLineBuilder] io.quarkus:quarkus-bootstrap-maven-resolver::jar classpath entry /Users/orion/.m2/repository/io/quarkus/quarkus-bootstrap-maven-resolver/3.18.0/quarkus-bootstrap-maven-resolver-3.18.0.jar was overriden with /Users/orion/.m2/repository/io/quarkus/quarkus-bootstrap-maven-resolver/3.20.2.2/quarkus-bootstrap-maven-resolver-3.20.2.2.jar
WARNING: Using incubator modules: jdk.incubator.vector
Listening for transport dt_socket at address: 5005
2025-10-22 14:06:12,017 WARN [io.sma.config] (main) SRCFG01007: Duplicate value found for name : quarkus.log.category."uk.ac.manchester.tornado".level, config source name : PropertiesConfigSource[source=file:/Users/orion/quarkus-langchain4j/integration-tests/gpullama3/target/classes/application.properties], (old value : DEBUG)
2025-10-22 14:06:12,064 WARN [io.sma.config] (main) SRCFG01007: Duplicate value found for name : quarkus.log.category."uk.ac.manchester.tornado".level, config source name : PropertiesConfigSource[source=file:/Users/orion/quarkus-langchain4j/integration-tests/gpullama3/target/classes/application.properties], (old value : DEBUG)
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2025-10-22 14:06:12,674 WARN [io.sma.config] (Quarkus Main Thread) SRCFG01007: Duplicate value found for name : quarkus.log.category."uk.ac.manchester.tornado".level, config source name : PropertiesConfigSource[source=file:/Users/orion/quarkus-langchain4j/integration-tests/gpullama3/target/classes/application.properties], (old value : DEBUG)
2025-10-22 14:06:12,770 WARN [io.sma.config] (Quarkus Main Thread) SRCFG01007: Duplicate value found for name : quarkus.log.category."uk.ac.manchester.tornado".level, config source name : PropertiesConfigSource[source=file:/Users/orion/quarkus-langchain4j/integration-tests/gpullama3/target/classes/application.properties], (old value : DEBUG)
2025-10-22 14:06:12,787 WARN [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.log.syslog.enabled" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2025-10-22 14:06:12,787 WARN [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.package.jvm-args" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2025-10-22 14:06:12,788 WARN [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.class-loading.parent-first-packages" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2025-10-22 14:06:12,788 WARN [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.log.console.enabled" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2025-10-22 14:06:12,824 INFO [io.qua.lan.gpu.run.GPULlama3Recorder] (Quarkus Main Thread) Set module paths: .:/Users/orion/TornadoVM/bin/sdk/share/java/tornado and upgrade path: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars
2025-10-22 14:06:12,826 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Safe initialization of TornadoVM
2025-10-22 14:06:12,826 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) === TornadoVM Environment Information ===
2025-10-22 14:06:12,826 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) TORNADO_SDK: /Users/orion/TornadoVM/bin/sdk
2025-10-22 14:06:12,826 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) SDK directory exists: true
2025-10-22 14:06:12,826 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) bin directory: /Users/orion/TornadoVM/bin, exists: true
2025-10-22 14:06:12,827 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) tornado command: /Users/orion/TornadoVM/bin/tornado, exists: false
2025-10-22 14:06:12,827 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) lib directory: /Users/orion/TornadoVM/bin/sdk/lib, exists: true
2025-10-22 14:06:12,827 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) tornado JARs directory: /Users/orion/TornadoVM/bin/sdk/share/java/tornado, exists: true
2025-10-22 14:06:12,828 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) graal JARs directory: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars, exists: true
2025-10-22 14:06:12,828 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) java.library.path: /Users/orion/TornadoVM/bin/sdk/lib
2025-10-22 14:06:12,828 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) jdk.module.path: .:/Users/orion/TornadoVM/bin/sdk/share/java/tornado
2025-10-22 14:06:12,828 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) jdk.module.upgrade.path: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars
2025-10-22 14:06:12,828 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Current class loader: QuarkusClassLoader:Quarkus Runtime ClassLoader: DEV for quarkus-langchain4j-integration-test-gpu-llama3-999-SNAPSHOT restart no:0@203d1d93
2025-10-22 14:06:12,829 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) tornado.load.api.implementation = uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph
2025-10-22 14:06:12,829 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) tornado.load.annotation.implementation = uk.ac.manchester.tornado.annotation.ASMClassVisitor
2025-10-22 14:06:12,829 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) tornado.load.annotation.parallel = uk.ac.manchester.tornado.api.annotations.Parallel
2025-10-22 14:06:12,829 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) tornado.load.tornado.implementation = uk.ac.manchester.tornado.runtime.common.Tornado
2025-10-22 14:06:12,829 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) tornado.load.runtime.implementation = uk.ac.manchester.tornado.runtime.TornadoCoreRuntime
2025-10-22 14:06:12,829 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) === configureLibraryPath() ===
2025-10-22 14:06:12,830 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Libraries found in /Users/orion/TornadoVM/bin/sdk/lib:
2025-10-22 14:06:12,830 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) - libtornado-opencl.dylib
2025-10-22 14:06:12,830 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) === Adjusting class loaders for TornadoVM ===
2025-10-22 14:06:12,830 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Found QuarkusClassLoader: QuarkusClassLoader:Quarkus Runtime ClassLoader: DEV for quarkus-langchain4j-integration-test-gpu-llama3-999-SNAPSHOT restart no:0@203d1d93
2025-10-22 14:06:12,830 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: uk.ac.manchester.tornado
2025-10-22 14:06:12,831 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: org.graalvm
2025-10-22 14:06:12,831 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: uk.ac.manchester.tornado.api
2025-10-22 14:06:12,831 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: uk.ac.manchester.tornado.api.runtime
2025-10-22 14:06:12,831 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: uk.ac.manchester.tornado.api.common
2025-10-22 14:06:12,831 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: uk.ac.manchester.tornado.runtime
2025-10-22 14:06:12,831 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: uk.ac.manchester.tornado.drivers
2025-10-22 14:06:12,831 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: uk.ac.manchester.tornado.drivers.opencl
2025-10-22 14:06:12,832 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: jdk.internal.vm.compiler
2025-10-22 14:06:12,832 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added package to parent-first: jdk.internal.vm.ci
2025-10-22 14:06:12,832 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Added TornadoVM packages to parent-first set
2025-10-22 14:06:12,832 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Tornado JAR path: /Users/orion/TornadoVM/bin/sdk/share/java/tornado
2025-10-22 14:06:12,832 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Graal JAR path: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars
2025-10-22 14:06:12,833 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Tornado JAR directory: /Users/orion/TornadoVM/bin/sdk/share/java/tornado, exists: true, is dir: true
2025-10-22 14:06:12,833 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Graal JAR directory: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars, exists: true, is dir: true
2025-10-22 14:06:12,833 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/onnxruntime-1.17.1.jar
2025-10-22 14:06:12,833 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/commons-lang3-3.18.0.jar
2025-10-22 14:06:12,833 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/ejml-simple-0.38.jar
2025-10-22 14:06:12,833 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-api-1.1.2-dev.jar
2025-10-22 14:06:12,833 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-benchmarks-1.1.2-dev.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/jmh-core-1.29.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/ejml-core-0.38.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-drivers-opencl-jni-1.1.2-dev-libs.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-matrices-1.1.2-dev.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/log4j-api-2.17.1.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/nativeimage-23.1.0.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/asm-9.5.jar
2025-10-22 14:06:12,834 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/junit-4.13.2.jar
2025-10-22 14:06:12,835 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/ejml-dsparse-0.38.jar
2025-10-22 14:06:12,835 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/log4j-core-2.17.1.jar
2025-10-22 14:06:12,835 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/ejml-ddense-0.38.jar
2025-10-22 14:06:12,835 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/word-23.1.0.jar
2025-10-22 14:06:12,835 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/lucene-core-8.2.0.jar
2025-10-22 14:06:12,835 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-examples-1.1.2-dev.jar
2025-10-22 14:06:12,835 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/commons-math3-3.6.1.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-runtime-1.1.2-dev.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-annotation-1.1.2-dev.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-unittests-1.1.2-dev.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/hamcrest-core-1.3.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/collections-23.1.0.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/snmp4j-2.8.6.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-drivers-common-1.1.2-dev.jar
2025-10-22 14:06:12,836 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/polyglot-23.1.0.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/jopt-simple-4.6.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/jsr305-3.0.2.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/tornado/tornado-drivers-opencl-1.1.2-dev.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/truffle-api-23.1.0.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/word-23.1.0.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/compiler-management-23.1.0.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/truffle-compiler-23.1.0.jar
2025-10-22 14:06:12,837 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/compiler-23.1.0.jar
2025-10-22 14:06:12,838 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/collections-23.1.0.jar
2025-10-22 14:06:12,838 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/graal-sdk-23.1.0.jar
2025-10-22 14:06:12,838 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Adding JAR to classpath: /Users/orion/TornadoVM/bin/sdk/share/java/graalJars/polyglot-23.1.0.jar
2025-10-22 14:06:12,838 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Set custom class loader as context loader temporarily
2025-10-22 14:06:12,856 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Got TornadoRuntime directly: uk.ac.manchester.tornado.runtime.TornadoCoreRuntime@38365401
2025-10-22 14:06:12,856 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Number of backends: 0
2025-10-22 14:06:12,856 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Restored original context class loader
2025-10-22 14:06:12,856 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Restored original context class loader
2025-10-22 14:06:12,857 WARN [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) No backends available in TornadoRuntime
2025-10-22 14:06:12,857 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) TornadoVM initialized: false
2025-10-22 14:06:12,857 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Checking TornadoVM installation using external process
2025-10-22 14:06:12,857 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Checking tornado command at: /Users/orion/TornadoVM/bin/tornado, exists: false
2025-10-22 14:06:12,857 WARN [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) Tornado command not found at: /Users/orion/TornadoVM/bin/tornado or /Users/orion/TornadoVM/bin/tornado
2025-10-22 14:06:12,857 INFO [io.qua.lan.gpu.run.TornadoVMSafety] (Quarkus Main Thread) External TornadoVM check: false
2025-10-22 14:06:12,857 INFO [io.qua.lan.gpu.run.GPULlama3Recorder] (Quarkus Main Thread) TornadoVM runtime initialized (though it may have 0 backends)
2025-10-22 14:06:12,915 INFO [io.quarkus] (Quarkus Main Thread) quarkus-langchain4j-integration-test-gpu-llama3 999-SNAPSHOT on JVM (powered by Quarkus 3.20.2.2) started in 1.002s. Listening on: http://localhost:8080
2025-10-22 14:06:12,916 INFO [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2025-10-22 14:06:12,916 INFO [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, langchain4j, langchain4j-gpu-llama3, qute, rest, rest-jackson, smallrye-context-propagation, vertx]when curl: 2025-10-22 14:06:36,482 ERROR [io.qua.lan.gpu.run.GPULlama3Recorder] (executor-thread-1) Error creating GPULlama3ChatModel
Exception in ChatLanguageModelResource.java:20
18 @Path("blocking")
19 public String blocking() {
→ 20 return chatModel.chat("When was the nobel prize for economics first awarded?");
21 }
22 }
: java.lang.NullPointerException: Cannot invoke "uk.ac.manchester.tornado.api.TornadoBackend.getDefaultDevice()" because the return value of "uk.ac.manchester.tornado.api.TornadoRuntime.getBackend(int)" is null
at org.beehive.gpullama3.tornadovm.TornadoVMMasterPlan.shouldUseNvidiaScheduler(TornadoVMMasterPlan.java:122)
at org.beehive.gpullama3.tornadovm.TornadoVMMasterPlan.<init>(TornadoVMMasterPlan.java:32)
at org.beehive.gpullama3.tornadovm.TornadoVMMasterPlan.initializeTornadoVMPlan(TornadoVMMasterPlan.java:64)
at io.quarkiverse.langchain4j.gpullama3.GPULlama3BaseModel.init(GPULlama3BaseModel.java:52)
at io.quarkiverse.langchain4j.gpullama3.GPULlama3ChatModel.init(GPULlama3ChatModel.java:15)
at io.quarkiverse.langchain4j.gpullama3.GPULlama3ChatModel.<init>(GPULlama3ChatModel.java:19)
at io.quarkiverse.langchain4j.gpullama3.GPULlama3ChatModel$Builder.build(GPULlama3ChatModel.java:103)
at io.quarkiverse.langchain4j.gpullama3.runtime.GPULlama3Recorder.lambda$chatModel$0(GPULlama3Recorder.java:106)
at io.quarkus.arc.runtime.ArcRecorder$4.apply(ArcRecorder.java:137)
at io.quarkus.arc.runtime.ArcRecorder$4.apply(ArcRecorder.java:134)
at dev.langchain4j.model.chat.ChatModel_CAOdzSz_JXOmbdPru8SBVdBro64_Synthetic_Bean.createSynthetic(Unknown Source)
at dev.langchain4j.model.chat.ChatModel_CAOdzSz_JXOmbdPru8SBVdBro64_Synthetic_Bean.doCreate(Unknown Source)
at dev.langchain4j.model.chat.ChatModel_CAOdzSz_JXOmbdPru8SBVdBro64_Synthetic_Bean.create(Unknown Source)
at dev.langchain4j.model.chat.ChatModel_CAOdzSz_JXOmbdPru8SBVdBro64_Synthetic_Bean.create(Unknown Source)
at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c24(Unknown Source)
at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:23)
at dev.langchain4j.model.chat.ChatModel_CAOdzSz_JXOmbdPru8SBVdBro64_Synthetic_ClientProxy.arc$delegate(Unknown Source)
at dev.langchain4j.model.chat.ChatModel_CAOdzSz_JXOmbdPru8SBVdBro64_Synthetic_ClientProxy.chat(Unknown Source)
at org.acme.example.gpullama3.chat.ChatLanguageModelResource.blocking(ChatLanguageModelResource.java:20)
at org.acme.example.gpullama3.chat.ChatLanguageModelResource$quarkusrestinvoker$blocking_ca0ebb7a906548c4afb53487d7744ae645968852.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:638)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
|
|
Thanks! I was going to suggest using |
I just updated the snippets above. |
|
with this fix, Quarkus is able to successfully load and init tornado runtime and detect its backends: and if we trigger the integrated test endpoint we successfully get response from the gpullama3: |
|
Thanks @geoand ! All works now, feel free to test it. |
|
Awesome! I will update the PR on Friday to make that classloading property unnecessary (by making the extension set it automatically) |
orionpapadakis
force-pushed
the
gpu-llama3-integration
branch
from
6aa8244 to
31a0ca3
Compare
geoand
force-pushed
the
gpu-llama3-integration
branch
2 times, most recently
from
25bfb35 to
d160109
Compare
|
Thanks a ton for this work folks! I am having trouble building TornadoVM locally, so I'll have to rely on you to test my updates (essentially making the proper class loading config part of the extension, not the application). |
|
I can confirm it works in both dev and production mode: dev:
production:
|
if you can share the issue/trace we can see if we can help with local installation. Usually, the /tornadovm-installer is suffiecient if all drivers for nvidia are installed properly in the system. |
|
This is what Maven is showing: Note that neither |
is what I have on my Ubuntu 24.04 system |
can you share the whole log because it should be more info? I suspect if you are installing for OpenCL backend you might miss the opencl-headers sudo apt install opencl-headers
sudo apt install ocl-icd-opencl-dev |
|
Thanks! Now it seems to work! |
|
So, here is what I see when trying to run the integration-test: We should move |
This comment has been minimized.
This comment has been minimized.
Shall we follow a similar approach as |
|
Right, I thought of that after writing the above comment. Would you like to go ahead and add that? |
Yes sure! |
|
🙏🏽 |
|
Currently our implementation supports only chat model. We are adding support for streaming. Would you prefer to include it in this PR or at a next one ? |
|
We can do that in a follow up |
This comment has been minimized.
This comment has been minimized.
|
Oh, I forgot I need to disable the tests in CI as Tornado VM is not available. |
geoand
force-pushed
the
gpu-llama3-integration
branch
2 times, most recently
from
b7560ad to
3f0ae65
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
geoand
force-pushed
the
gpu-llama3-integration
branch
2 times, most recently
from
9666dad to
f5eeef1
Compare
This comment has been minimized.
This comment has been minimized.
geoand
force-pushed
the
gpu-llama3-integration
branch
2 times, most recently
from
1262958 to
cd80fe6
Compare
This comment has been minimized.
This comment has been minimized.
geoand
force-pushed
the
gpu-llama3-integration
branch
from
cd80fe6 to
639637a
Compare
This comment has been minimized.
This comment has been minimized.
geoand
force-pushed
the
gpu-llama3-integration
branch
from
639637a to
3563933
Compare
This comment has been minimized.
This comment has been minimized.
geoand
force-pushed
the
gpu-llama3-integration
branch
from
3563933 to
e712762
Compare
This comment has been minimized.
This comment has been minimized.
Status for workflow
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR integrates GPULlama3.java to Quarkus Langchain4j extension as an additional model provider.
The successful deployment of this feature requires a system with a GPU, a local build of TornadoVM (detailed instructions are provided in model-providers/gpu-llama3/README.md) , a local copy of a GPULlama3.java compatible LLM in gguf format.
A set of models tested with GPULlama3.java is here:
https://huggingface.co/beehive-lab/collections