feat(loadIndices): loadIndices upgrade

[david-leifker]

LoadIndices Upgrade - High-Performance Bulk Metadata Loading

Documentation: https://github.com/datahub-project/datahub/blob/45a2fb3e2236540c31568984a92d3ba98fc15a9f/docs/how/load-indices.md

🎯 Feature Overview

This PR introduces LoadIndices, a high-performance upgrade system designed for bulk loading metadata aspects directly from the database into Elasticsearch/OpenSearch indices. Unlike traditional restore operations that prioritize correctness, LoadIndices is optimized for speed and throughput during initial deployments and large-scale data migrations.

🚀 Core Capabilities

High-Performance Bulk Loading

• Direct database streaming from metadata_aspect_v2 table to search indices
• Optimized bulk operations with automatic refresh interval management
• Bypasses Kafka MCL pipeline for maximum performance during bulk loads
• Configurable transaction isolation using READ_UNCOMMITTED for faster scanning

Intelligent Index Management

• Automatic refresh control - disables refresh intervals during bulk loading for optimal performance
• Smart index optimization - re-enables refresh intervals after completion
• Comprehensive progress monitoring with real-time reporting and performance metrics
• Elasticsearch/OpenSearch compatibility with optimized bulk write operations

Enterprise Integration

• Spring Boot integration with conditional configuration based on command-line arguments
• Production-ready architecture with proper error handling and recovery mechanisms
• Comprehensive documentation with usage guidelines and performance considerations

⚡ Performance Benefits

• Significantly faster than traditional restore operations for large datasets
• Optimized for initial deployments and data center migrations
• Bulk operation efficiency with minimal overhead and intelligent request distribution
• Scalable architecture designed for enterprise-scale metadata volumes

📋 Key Use Cases

• Initial DataHub deployments with large existing metadata
• Data center migrations requiring bulk index population
• Disaster recovery scenarios with large-scale data restoration
• Development environment setup with comprehensive test data

⚠️ Design Philosophy

LoadIndices makes strategic architectural trade-offs to prioritize performance over consistency:

• Bypasses Kafka MCL event pipeline for direct database-to-index streaming
• Uses READ_UNCOMMITTED transactions for faster database scanning
• Optimized for bulk operations rather than real-time consistency

This feature enables organizations to efficiently populate their DataHub search indices during initial setup or large-scale migrations, providing a fast path to getting DataHub operational with comprehensive metadata search capabilities and enterprise-grade performance.

[codecov]

Codecov Report

❌ Patch coverage is 77.89905% with 162 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...n/datahub/upgrade/loadindices/LoadIndicesStep.java	83.82%	36 Missing and 13 partials ⚠️
...b/upgrade/loadindices/LoadIndicesIndexManager.java	75.24%	22 Missing and 3 partials ⚠️
.../java/com/linkedin/datahub/upgrade/UpgradeCli.java	0.00%	17 Missing ⚠️
...ub/upgrade/loadindices/NoOpKafkaEventProducer.java	56.41%	17 Missing ⚠️
...icsearch/client/shim/impl/Es8SearchClientShim.java	81.81%	9 Missing and 1 partial ⚠️
...inkedin/metadata/service/UpdateIndicesService.java	25.00%	9 Missing ⚠️
...a/search/elasticsearch/update/ESBulkProcessor.java	46.15%	7 Missing ⚠️
...rch/elasticsearch/indexbuilder/ESIndexBuilder.java	0.00%	4 Missing ⚠️
...e/system/elasticsearch/steps/ReindexDebugStep.java	66.66%	2 Missing and 1 partial ⚠️
...ch/client/shim/impl/AbstractBulkProcessorShim.java	86.36%	3 Missing ⚠️
... and 10 more

📢 Thoughts on this report? Let us know!

[codecov]

Bundle Report

Bundle size has no change ✅

[alwaysmeticulous]

✅ Meticulous spotted 0 visual differences across 971 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit 7f486f9. This comment will update as new commits are pushed.}