Clarification of the scripts caching vs reuse of context

[and-za]

Hi. I probably does not understand things around scripts caching well enough. Here is my "problem".

I have a very simple JavaScript script which just concatenates two strings. When I call this script in a loop of lets say 100 000 iterations, and in each iteration I use the same Context, then it's pretty quick.

But when I want to use "engine sharing" instead, that means in each iteration create new Context but with shared Engine, then it is very slow.

My understanding was, that the script evaluated in first (few) iterations would be parsed, optimized and cached in the engine, so each subsequent context would benefit from the optimizations/parsing in the first (few) iterations. But that does not seem to be true. Could you please clarify to me, how is it intended to work and why there is so big difference?

I have tested it with OpenJDK 21, but also with Graal VM 21/24. The behavior is more or less the same. Graal version 23.1.2, but also the 24.2.1 on JDK 24.

Here is the code which I used to "benchmark" it:

public class GraalJsExample {

    private static final Source SCRIPT = Source.create("js", "param1 + param2");

    public static void main(String[] args) {
        System.out.println("Generating random strings...");
        final var strings = createRandomStrings(100_001);

        System.out.println("Benchmarking");
        for (int i = 0; i < 10; i++) {
            System.out.println("Iteration " + (i + 1) + " ---------------------");
            final long startLoop = System.nanoTime();
            // Uncomment one of bellow two lines
            benchmarkWithContextReuse(strings);
            //benchmarkWithSharedEngine(strings);
            final long endLoop = System.nanoTime();

            final double totalSeconds = (endLoop - startLoop) / 1_000_000_000.0;
            System.out.printf("JS execution loop completed in: %.3f seconds%n", totalSeconds);
            System.out.printf("Average time per iteration: %.3f ms%n",
                    (endLoop - startLoop) / 1_000_000.0 / (strings.size() -1));
        }
    }

    private static void benchmarkWithContextReuse(List<String> strings) {
        int charactersCount = 0;
        try (final Context context = Context.newBuilder()
                .allowHostAccess(HostAccess.UNTRUSTED)
                .build()) {
            for (int i = 1; i < strings.size(); i++) {
                charactersCount += concatWithJs(context, strings.get(i - 1), strings.get(i)).length();
            }

        }
        System.out.printf("Number of characters (Just to use the script result for something): %d.%n", charactersCount);
    }

    private static void benchmarkWithSharedEngine(List<String> strings) {
        int charactersCount = 0;
        try (final Engine engine = Engine.create()) {
            for (int i = 1; i < strings.size(); i++) {
                try (final Context context = Context.newBuilder()
                        .engine(engine)
                        .allowHostAccess(HostAccess.UNTRUSTED)
                        .build()) {

                    charactersCount += concatWithJs(context, strings.get(i - 1), strings.get(i)).length();
                }
            }
        }

        System.out.printf("Number of characters (Just to use the script result for something): %d.%n", charactersCount);
    }

    private static String concatWithJs(Context context, String param1, String param2) {
            context.getBindings("js").putMember("param1", param1);
            context.getBindings("js").putMember("param2", param2);
            return context.eval(GraalJsExample.SCRIPT).asString();
    }

    private static String concatWithGroovy(Script script, String param1, String param2) {
        final Binding binding = new Binding(Map.of("param1", param1, "param2", param2));
        script.setBinding(binding);
        return script.run().toString();
    }

    private static List<String> createRandomStrings(int count) {
        final Random random = new Random();
        final List<String> result = new ArrayList<>(count);

        // Generate random strings of length 5-15 characters
        for (int i = 0; i < count; i++) {
            final int length = random.nextInt(11) + 5; // Random length between 5 and 15
            final StringBuilder sb = new StringBuilder(length);

            for (int j = 0; j < length; j++) {
                // Generate random printable ASCII characters (from '!' to '~')
                final char c = (char) (random.nextInt(94) + 33);
                sb.append(c);
            }

            result.add(sb.toString());
        }

        return result;
    }
}

[woess]

Given that your js code snippet is trivial, I suspect that you're measuring mostly Context initialization overhead and thus not seeing the benefits of code caching. Context creation isn't free, but usually dominated by the JS code.
If your JS code is very short-running and you don't need isolated contexts, I would suggest to consider reusing thread-local contexts.

[chumer]

If you can't reuse the context, as the global state of the scripts should not be shared, then another option is to share an engine across contexts. There are some docs here:
https://www.graalvm.org/latest/reference-manual/embed-languages/#code-caching-across-multiple-contexts

[and-za]

Thanks for your replies.

I suspect that you're measuring mostly Context initialization overhead and thus not seeing the benefits of code caching

I have made few more measurements. Please take them with grain of salt. I am quite new to such bench-marking, so the benchmarks probably have their flaws.

Benchmark                                      Mode  Cnt  Score    Error  Units
# Baseline, which does almost the same thing as I described in original post.
concatenationUsingSeparateContexts             avgt    5  0.109 ±  0.003  ms/op
# This one creates, binds variables and then close context
contextCreationAndClosingPlusVariableBindings  avgt    5  0.106 ±  0.003  ms/op
# Just creates and closes context
contextCreationAndClosing                      avgt    5  0.002 ±  0.001  ms/op
# Bind variables to new context
variableBindingsToNewContext                   avgt    5  0.101 ±  0.005  ms/op
# Rewrite existing variables in existing context
rewritingBindingsInSameContext                 avgt    5  0.001 ±  0.001  ms/op

From my measurements it looks like the costly operation is not the initialization itself, but the binding of variables. That aligns also with several bug reports (oracle/graaljs#788, oracle/graaljs#561). Interestingly, the "rewriting" of existing bindings in existing context is significantly faster then binding variables to new context.

I assume you are already aware of that, but I thought that I will write it all here just for some future readers with similar issues.

@chumer I already share the engine, so that will not improve things. And I guess that it would not help with bindings anyway (correct me if I am wrong).

Here is the benchmarks code:

package com.evolveum.scripting;

import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.concurrent.TimeUnit;

import org.graalvm.polyglot.Context;
import org.graalvm.polyglot.Engine;
import org.graalvm.polyglot.HostAccess;
import org.graalvm.polyglot.Source;
import org.graalvm.polyglot.Value;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.TearDown;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.infra.Blackhole;

@Warmup(iterations = 3, time = 10, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 10, timeUnit = TimeUnit.SECONDS)
@Fork(value = 1)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
public class GraalJsContextBenchmark {
    private static final String SCRIPT = "param1 + param2";
    private static final int STRING_COUNT = 100001;

    private List<String> data;
    private Source graalJsSource;
    private Engine graalSharedEngine;
    private Context graalSharedContext;
    private int counter = 0;

    @Setup(Level.Trial)
    public void setUp() {
        this.data = createRandomStrings(STRING_COUNT);
        this.graalJsSource = Source.create("js", SCRIPT);
        this.graalSharedContext = Context.newBuilder()
                .allowHostAccess(HostAccess.UNTRUSTED)
                .build();
        this.graalSharedEngine = Engine.create();
    }

    @TearDown(Level.Trial)
    public void tearDown() {
        this.graalSharedContext.close();
        this.graalSharedEngine.close();
    }

    @Benchmark
    public void concatenationUsingSeparateContexts(Blackhole blackHole) {
        try (final Context context = Context.newBuilder()
                .engine(this.graalSharedEngine)
                .allowHostAccess(HostAccess.UNTRUSTED)
                .build()) {
            context.getBindings("js").putMember("param1", this.data.get(this.counter));
            context.getBindings("js").putMember("param2", this.data.get(this.counter++));
            blackHole.consume(context.eval(this.graalJsSource).asString().length());
        }
        if (this.counter == STRING_COUNT) {
            this.counter = 0;
        }
    }

    @Benchmark
    public void contextCreationAndClosing(Blackhole blackhole) {
        try (final Context context = Context.newBuilder()
                .engine(this.graalSharedEngine)
                .allowHostAccess(HostAccess.UNTRUSTED)
                .build()) {
            // I am not sure if this is strictly needed, I had the same measurement with and without it
            blackhole.consume(context);
        }
    }

    @Benchmark
    public void contextCreationAndClosingPlusVariableBindings(Blackhole blackhole) {
        try (final Context context = Context.newBuilder()
                .engine(this.graalSharedEngine)
                .allowHostAccess(HostAccess.UNTRUSTED)
                .build()) {
            final Value bindings = context.getBindings("js");
            bindings.putMember("param1", this.data.get(this.counter));
            bindings.putMember("param1", this.data.get(this.counter++));
            blackhole.consume(bindings);
        }
        if (this.counter == STRING_COUNT) {
            this.counter = 0;
        }
    }

    @Benchmark
    public void variableBindingsToNewContext(Blackhole blackhole, ContextPerInvocation contextPerInvocation) {
        final Value bindings = contextPerInvocation.context.getBindings("js");
        bindings.putMember("param1", this.data.get(this.counter));
        bindings.putMember("param1", this.data.get(this.counter++));
        blackhole.consume(bindings);
        if (this.counter == STRING_COUNT) {
            this.counter = 0;
        }
    }

    @Benchmark
    public void rewritingBindingsInSameContext(Blackhole blackhole) {
        final Value bindings = this.graalSharedContext.getBindings("js");
        bindings.putMember("param1", this.data.get(this.counter));
        bindings.putMember("param1", this.data.get(this.counter++));
        blackhole.consume(bindings);
        if (this.counter == STRING_COUNT) {
            this.counter = 0;
        }
    }

    @State(Scope.Benchmark)
    public static class ContextPerInvocation {
        private Engine engine;
        Context context;

        @Setup(Level.Trial)
        public void setupOnce() {
            this.engine = Engine.create();
        }

        @TearDown(Level.Trial)
        public void teardownOnce() {
            this.engine.close();
        }

        @Setup(Level.Invocation)
        public void setup() {
            this.context = Context.newBuilder()
                    .engine(this.engine)
                    .allowHostAccess(HostAccess.UNTRUSTED)
                    .build();
        }

        @TearDown(Level.Invocation)
        public void teardown() {
            this.context.close();
        }
    }

    private static List<String> createRandomStrings(int count) {
        final Random random = new Random();
        final List<String> result = new ArrayList<>(count);
        for (int i = 0; i < count; i++) {
            final int length = random.nextInt(11) + 5; // Random length between 5 and 15
            final StringBuilder sb = new StringBuilder(length);
            for (int j = 0; j < length; j++) {
                final char c = (char) (random.nextInt(94) + 33); // '!'..'~'
                sb.append(c);
            }
            result.add(sb.toString());
        }
        return result;
    }
}

[woess]

From my measurements it looks like the costly operation is not the initialization itself, but the binding of variables.

I don't think this is true. As you've observed:

Interestingly, the "rewriting" of existing bindings in existing context is significantly faster then binding variables to new context.

While adding new bindings can be a bit slower than writing existing bindings, it should not be that much slower.
What's actually going on here* is that the contextCreationAndClosing benchmark is not actually initializing the js language context, because that happens lazily on the first eval—or—bindings access. FWIW, you can force initialization using context.initialize("js"), in which case you should see similar ms/op compared to variableBindingsToNewContext.

* Disclaimer: I haven't actually run the benchmarks myself yet, so this is more of an educated guess.

[and-za]

Thanks for the insight. Indeed, I run the benchmarks again with the forced initialization and you are right:

Benchmark                                      Mode  Cnt  Score   Error  Units
contextCreationAndClosing                      avgt    5  0.105 ± 0.005  ms/op
contextCreationAndClosingPlusVariableBindings  avgt    5  0.101 ± 0.004  ms/op

Interestingly, the variable bindings is a little bit faster then forced initialization alone, but it is so small difference, that it probably could be caused by benchmark itself or by warmup or I don't know what.