Skip to content

Prevent auto-sharding for data streams in LOOKUP index mode #131429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

lukewhiting
Copy link
Contributor

This PR disabled the datastream autosharding for LOOKUP indices to prevent them scaling above 1 replica which is unsupported by the lookup mappers.

Fixes ES-12330

@lukewhiting lukewhiting added >bug :Data Management/Data streams Data streams and their lifecycles auto-backport Automatically create backport pull requests when merged v9.0.0 v9.1.0 v9.2.0 v8.19.1 v8.18.5 labels Jul 17, 2025
@lukewhiting lukewhiting removed the request for review from PeteGillinElastic July 17, 2025 10:41
@elasticsearchmachine
Copy link
Collaborator

Hi @lukewhiting, I've created a changelog YAML for you.

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR prevents auto-sharding functionality for data streams that use LOOKUP index mode, as lookup mappers don't support scaling beyond 1 replica. The implementation adds an early return in the auto-sharding calculation logic when the index mode is LOOKUP.

  • Adds a check in the auto-sharding service to return NOT_APPLICABLE_RESULT for LOOKUP index mode data streams
  • Includes comprehensive test coverage for both scenarios with and without index statistics

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
DataStreamAutoShardingService.java Adds LOOKUP index mode check to prevent auto-sharding calculation
DataStreamAutoShardingServiceTests.java Adds test cases to verify auto-sharding is disabled for LOOKUP index mode
Comments suppressed due to low confidence (2)

server/src/test/java/org/elasticsearch/action/datastreams/autosharding/DataStreamAutoShardingServiceTests.java:1347

  • [nitpick] The test method name could be more descriptive. Consider renaming to 'testCalculateReturnsNotApplicableForLookupIndexModeWithStats' to better distinguish it from the null stats test case.
    public void testCalculateReturnsNotApplicableForLookupIndexMode() {

.putProjectMetadata(builder.build())
.build();

AutoShardingResult autoShardingResult = service.calculate(
Copy link
Preview

Copilot AI Jul 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's code duplication between the two test methods. Consider extracting the common DataStream and ClusterState setup into a helper method to improve maintainability.

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Automatically create backport pull requests when merged >bug :Data Management/Data streams Data streams and their lifecycles v8.18.5 v8.19.1 v9.0.0 v9.1.0 v9.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants