Janusgraph upgrade to 1.1.0

[Shardul-Bh]

After upgrading to Janusgraph 1.1.0, I am started getting below issue

Cannot read the array length because "elementData" is null\n\tat java.base/java.util.ArrayList$Itr.next(ArrayList.java:972)\n\tat org.janusgraph.graphdb.database.IndexSerializer.getIndexUpdates(IndexSerializer.java:257)\n\tat org.janusgraph.graphdb.database.IndexSerializer.getIndexUpdates(IndexSerializer.java:206)\n\tat org.janusgraph.graphdb.database.StandardJanusGraph.lambda$prepareCommitVertexIndexUpdates$2(StandardJanusGraph.java:819)\n\tat

I have added below property but its still getting same
storage.num-mutations-parallel-threshold value as 1000 . This is coming on high transaction ..Please suggest

[Shardul-Bh]

Please find the more details
NullPointerException in prepareCommitVertexIndexUpdates after upgrading to 1.1.0
Environment

JanusGraph Version: 1.1.0
Storage Backend: Cassandra (CQL)
Previous Working Version: 1.0.0
Java Version: 17 (based on stack trace)
Spring Boot: Yes
Deployment: Kubernetes

Problem Description
After upgrading from JanusGraph 1.0.0 to 1.1.0, we are experiencing intermittent NullPointerException during transaction commits in performance testing. The issue occurs in prepareCommitVertexIndexUpdates() during parallel stream processing.
Error Message
java.lang.NullPointerException: Cannot read the array length because "elementData" is null
at java.base/java.util.ArrayList$Itr.next(ArrayList.java:972)
at org.janusgraph.graphdb.database.IndexSerializer.getIndexUpdates(IndexSerializer.java:257)
at org.janusgraph.graphdb.database.IndexSerializer.getIndexUpdates(IndexSerializer.java:206)
at org.janusgraph.graphdb.database.StandardJanusGraph.lambda$prepareCommitVertexIndexUpdates$2(StandardJanusGraph.java:819)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:960)
at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:934)
at java.base/java.util.stream.AbstractTask.compute(AbstractTask.java:327)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:927)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommitVertexIndexUpdates(StandardJanusGraph.java:820)
at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommit(StandardJanusGraph.java:726)
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:993)
Full Stack Trace

Workload Characteristics

Transaction Rate: 16 TPS
Transaction Size: 6 updated/create per transaction
Issue Manifestation: Intermittent crashes during performance testing

What We've Tried

1. Configuration Change
   We added the following configuration based on PR Optimization for adding new vertex and updating vertex properties #4292:
   propertiesstorage.num-mutations-parallel-threshold=1000
   Result: Exception frequency reduced but NOT eliminated

Before: Frequent NPEs
After: Occasional NPEs (less frequent but still occurring)

2. Increasing Threshold Further
   We're considering increasing the threshold to 10000 or higher, but unsure if this will help.
   Questions

What does storage.num-mutations-parallel-threshold actually control?

Does it affect only the consolidate() method?
Does it affect prepareCommitVertexIndexUpdates() as well?
Why does increasing it reduce but not eliminate the issue?

Is this a known issue in JanusGraph 1.1.0?

We suspect this is related to PR #4292 which introduced parallel processing optimizations

What is the recommended fix?

Should we increase the threshold further?
Should we downgrade to 1.0.0?
Is there another configuration option we're missing?
Will this be fixed in 1.1.1 or 1.2.0?

Root Cause Analysis (Our Understanding)
The error message Cannot read the array length because "elementData" is null at ArrayList$Itr.next(ArrayList.java:972) suggests a race condition where:

Multiple threads from parallelStream() are accessing the same ArrayList instance
One thread is iterating while another thread modifies/clears the list
The iterator tries to access elementData which has been set to null by another thread

This appears to be a concurrent modification issue in the parallel processing introduced in 1.1.0.
Expected Behavior
Transaction commits should complete successfully without NullPointerException, even under high concurrent load, as they did in version 1.0.0.
Actual Behavior
Intermittent NullPointerException in prepareCommitVertexIndexUpdates() causing transaction failures.
Impact

Severity: Critical - lead to data loss
Workaround: Currently none that fully resolves the issue
Production Impact: Blocking upgrade to 1.1.0

Additional Context

This issue did NOT occur in JanusGraph 1.0.0 with the same workload
The issue appears to be a race condition triggered by parallel stream processing
We are using Cassandra as the storage backend (not BerkeleyDB or other backends)

[manojpat-a]

@ntisseyre - can you please check and suggest, it seems the parallel-stream issue for for-each is being cascaded to internal for-loops used in calls from IndexSerializer.getIndexUpdates - causing this issue.

https://github.com/JanusGraph/janusgraph/commit/f3700c4b5ee4e92b431e8090086b519b187c79e0#diff-bf962b508e141a60293e6b9bdbb52a4dcd84e58c7fdd32f2e37f723dd90bceb6R133-R812

     /* Collect all index update for vertices
     */
    private void prepareCommitVertexIndexUpdates(final ListMultimap<InternalVertex, InternalRelation> mutatedProperties,
                                                 final List<IndexUpdate> indexUpdates){
        for (InternalVertex v : mutatedProperties.keySet()) {
            indexUpdates.addAll(indexSerializer.getIndexUpdates(v,mutatedProperties.get(v)));
        }
                                                 final List<IndexUpdate> indexUpdates) {
        mutatedProperties.keySet().parallelStream()
            .map(v -> indexSerializer.getIndexUpdates(v, mutatedProperties.get(v)))
            // Note: due to usage of parallel stream, the collector is used to synchronize insertions
            // into `indexUpdates` for thread safety reasons.
            // Using `forEach` directly isn't thread safe.
            .collect(Collectors.toList())
            .forEach(indexUpdates::addAll);
    }