Optimize relational tablet batch inserts

[jt2594838]

Description

This PR optimizes relational insertTablets handling in the Java Session client.

• Add a merge switch for relational tablets in Session builders and pools.
• Reuse column index maps when checking and merging relational tablets.
• Avoid unnecessary bitmap creation during merge.
• Copy tablet data column-by-column with array copy.
• Auto-disable merge after consecutive misses or when merge overhead is too high.
• Add an ignored integration test for manually comparing insertTablet and insertTablets performance as tablet count increases on the same table.

Tests

• mvn test -pl iotdb-client/session -am "-Dtest=SessionTest#testMergeRelationalTabletsWithHighDuplicatedColumns+testMergeRelationalTabletsSkipLowDuplicatedColumnsAndDifferentTables+testMergeRelationalTabletsStopAfterConsecutiveMisses+testMergeRelationalTabletsResetConsecutiveMissesAfterMerge+testMergeTabletsDisabledWhenMergeCostExceedsHalfOfInsertCost+testMergeTabletsKeepsEnabledWhenMergeCostIsNotTooHigh+testEnableMergeTabletsUsingBuilder" "-DfailIfNoTests=false" "-Dsurefire.failIfNoSpecifiedTests=false"
• mvn test-compile -pl integration-test -am -P with-integration-tests -DskipTests "-Drat.skip=true"
• mvn verify "-DskipUTs" "-Dit.test=IoTDBSessionRelationalIT#compareInsertTabletAndInsertTabletsPerformanceWithIncreasingTabletCount" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" -pl integration-test -am -PTableSimpleIT -P with-integration-tests

[Caideyipi]

I found a few issues that should be addressed before merging:

1. High: Relational insertTablets cannot auto-create a new table.

In InsertTablets.validateTableSchema, the call to validateTableSchema passes allowCreateTable=false. This makes ITableSession.insertTablets(List) fail when inserting into a new table, while the existing single-tablet insert path can auto-create the table. The current integration tests all create the tables first, so this behavior difference is not covered. Please either align this with the single-tablet path or clearly document and test that batch tablet insertion requires the table to exist.

2. Medium: The new insertRelationalTablets path bypasses table-model redirection.

Session.insertRelationalTablets always sends the request through getDefaultSessionConnection().insertTablets(request) and ignores RedirectException. The existing insertRelationalTablet path uses the table-model leader cache and groups rows/tablets by endpoint when redirection is enabled. In a cluster, this new path can keep sending requests to the default node and will not refresh tableModelDeviceIdToEndpoint, which may offset the intended performance gain. Please consider reusing or extending the existing relational tablet redirection logic for the batch path.

3. Low: Merge cost is recorded even when the insert fails.

recordMergeTabletsCost is called from finally, so failed RPCs or execution failures still contribute to the auto-disable decision. This can permanently disable merge due to failure-path latency rather than actual merge overhead. It would be safer to record the cost only after a successful insert, or explicitly exclude failed attempts from the heuristic.

Question: Is insertTablets intentionally not supposed to support auto table creation? If yes, the API documentation and tests should make that clear.

[Caideyipi]

Thanks for addressing the previous review comments. I re-checked the latest revision and there is still one blocking issue.

High: the new table-model insertTablets path can still drop data in cluster mode.

The current batch path catches RedirectException and only updates the leader cache via handleRelationalTabletsRedirection, but it does not retry the redirected tablets/rows. In addition, SessionConnection.insertTablets() throws the multi-device redirection as deviceEndPointMap, while the new handleRelationalTabletsRedirection only reads getEndPointList(), so in this case the table-model leader cache may not even be refreshed.

This is also visible in the current CI failure, and it hits the new/related tests directly:

• IoTDBSessionRelationalIT.insertTabletsAutoCreateTableTest: expected 40 rows but got 20.
• IoTDBInsertTableSessionPoolIT.testInsertTabletsWithDifferentTables: expected 5 rows but got 0.

Could you please make the batch insertTablets redirection handling complete? After receiving per-device/per-row redirect information, the client should regroup the affected tablets/rows by endpoint and retry the unwritten data, instead of only updating the cache and returning successfully.

[HTHou]

High: This creates one sub-Tablet per distinct device, while createEmptyRelationalTabletLike allocates every sub-tablet with the original tablet full rowSize at line 2974. An R-row tablet with D devices therefore allocates O(D * R * columns) storage, reaching O(R^2) when each row belongs to a different device. This also happens with an empty leader cache: all devices map to the default connection, but the inner map still creates D full-capacity tablets. This can exhaust the client heap before any RPC. Could we group rows once per endpoint, or pre-count rows per device and allocate exact capacities? A many-device stress test would also help.

[jt2594838]

Thanks for catching this. I changed the split logic to group rows directly by target endpoint and pre-count the rows for each endpoint before allocating sub-tablets. Each sub-tablet now has the exact required capacity, so the empty-cache case creates one R-row tablet instead of D full-capacity tablets. I also added a 1,000-distinct-device unit test that verifies only one exact-capacity tablet is allocated when all rows use the default endpoint. This will be included in the next push.

[HTHou]

Medium: Every successful merge replaces the accumulated candidate with a newly allocated tablet and recopies all previously accumulated rows at lines 3213-3224; constructing the wrapper then rescans those timestamps. Merging N compatible tablets is therefore quadratic in copying and allocation, which affects the exact workload this optimization targets. The adaptive cost check runs only after the expensive call and after ten samples, so it cannot protect the first large batch. Could we collect a compatible run and allocate and copy once, or impose an explicit merge-work cap? Please add a scaling test with a large mergeable list.

[jt2594838]

Agreed. I replaced the pairwise merge-and-recopy loop with compatible-run collection. The final schema and row count are accumulated first, then the merged tablet is allocated once and every source tablet is copied once. The old intermediate-tablet wrapper and pairwise merge implementation have been removed. I also added a scaling test with 5,000 compatible tablets, plus a test covering type conflicts on columns introduced by an earlier tablet in the run. This will be included in the next push.

[HTHou]

Medium: Although insertStatement is the InsertTablets AST wrapper, its inherited getDatabase method dynamically dispatches getInnerTreeStatement to the wrapped InsertMultiTabletsStatement. That statement getDatabaseName method scans every child, and AnalyzeUtils.getDatabaseName may invoke it twice. Calling it inside this N-tablet permission loop makes validation O(N^2) before schema work begins. Please resolve and unqualify the database once before the loop; repeated checks for the same table can also be deduplicated.

[jt2594838]

Fixed. The database is now resolved and unqualified once before the permission loop, and table names are deduplicated with a set so each table is checked only once. This removes the repeated InsertMultiTabletsStatement database scan and duplicate permission checks. This will be included in the next push.

[HTHou]

Medium: The batch redirection path cannot update the table leader cache as currently wired. TableModelPlanner.setRedirectInfo emits redirect metadata only for InsertTabletStatement, not the new InsertMultiTabletsStatement. In addition, SessionConnection.insertTablets uses verifySuccessWithRedirectionForMultiDevices, which creates a RedirectException with deviceEndPointMap keyed from request prefix paths (table names), while this handler reads only endPointList. Batch-only workloads therefore never learn or repair per-device leader routes and keep going through the default or stale DataNode. Please define redirect metadata aligned with the batch devices and add a test that verifies cache updates; successful REDIRECTION_RECOMMEND writes should not be retried.

[jt2594838]

Thanks, addressed on both sides. RelationalInsertMultiTabletsNode now concatenates the per-row redirect metadata produced by each relational tablet, and TableModelPlanner emits redirect sub-statuses for InsertMultiTabletsStatement as well. The session passes the row-aligned device IDs to the multi-device redirect verifier and consumes deviceEndPointMap to update tableModelDeviceIdToEndpoint. REDIRECTION_RECOMMEND is treated as a successful write: the client updates the cache without retrying. I added tests for server-side redirect-list aggregation and client-side cache updates/no retry. This will be included in the next push.