Skip to content

HIVE-29436: Iceberg materialized view rebuild locking issue #6330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kasakrisz wants to merge 3 commits into
base: master
Choose a base branch
from
+87 −18
Draft

HIVE-29436: Iceberg materialized view rebuild locking issue #6330

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6ee69c9
HIVE-29436: Iceberg materialized view rebuild locking issue
kasakrisz Feb 20, 2026
00e94e0
add test for subsequent MV rebuilds
kasakrisz Mar 10, 2026
8a458be
remove extra materialized view locking
kasakrisz Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_orc2.q
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,11 @@ select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
create materialized view mat1 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;

create materialized view mat2 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c <= 52;

select * from mat1;
select * from mat2;

alter materialized view mat1 disable rewrite;

Expand All @@ -38,6 +42,15 @@ insert into tbl_ice values (10, 'ten', 60);
explain cbo
alter materialized view mat1 rebuild;

alter materialized view mat1 rebuild;
alter materialized view mat2 rebuild;
Copy link

@amansinha100 amansinha100 Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this test we are checking if rebuild for different materialized views succeeds. Can we also test if rebuilds of the same MV also works both in a serialized way within a session and in a concurrent sessions scenario ?


select * from mat1;
select * from mat2;

-- test subsequent MV rebuilds
insert into tbl_ice values (20, 'twenty', 80);

alter materialized view mat1 rebuild;

select * from mat1;
74 changes: 74 additions & 0 deletions iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc2.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,22 @@ POSTHOOK: Output: default@mat1
POSTHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: Lineage: mat1.b SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:b, type:string, comment:null), ]
POSTHOOK: Lineage: mat1.c SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:c, type:int, comment:null), ]
PREHOOK: query: create materialized view mat2 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c <= 52
PREHOOK: type: CREATE_MATERIALIZED_VIEW
PREHOOK: Input: default@tbl_ice
PREHOOK: Output: database:default
PREHOOK: Output: default@mat2
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: create materialized view mat2 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c <= 52
POSTHOOK: type: CREATE_MATERIALIZED_VIEW
POSTHOOK: Input: default@tbl_ice
POSTHOOK: Output: database:default
POSTHOOK: Output: default@mat2
POSTHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: Lineage: mat2.b SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:b, type:string, comment:null), ]
POSTHOOK: Lineage: mat2.c SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:c, type:int, comment:null), ]
PREHOOK: query: select * from mat1
PREHOOK: type: QUERY
PREHOOK: Input: default@mat1
Expand All @@ -178,6 +194,17 @@ POSTHOOK: Input: default@mat1
POSTHOOK: Output: hdfs://### HDFS PATH ###
five 54
four 53
PREHOOK: query: select * from mat2
PREHOOK: type: QUERY
PREHOOK: Input: default@mat2
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: select * from mat2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@mat2
POSTHOOK: Output: hdfs://### HDFS PATH ###
one 50
three 52
two 51
PREHOOK: query: alter materialized view mat1 disable rewrite
PREHOOK: type: ALTER_MATERIALIZED_VIEW_REWRITE
PREHOOK: Input: default@mat1
Expand Down Expand Up @@ -248,6 +275,52 @@ HiveProject(tbl_ice.b=[$1], tbl_ice.c=[$2])
HiveFilter(condition=[>($2, 52)])
HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice], fromVersion=[#Masked#])

PREHOOK: query: alter materialized view mat1 rebuild
PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
PREHOOK: Input: default@tbl_ice
PREHOOK: Output: default@mat1
POSTHOOK: query: alter materialized view mat1 rebuild
POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
POSTHOOK: Input: default@tbl_ice
POSTHOOK: Output: default@mat1
PREHOOK: query: alter materialized view mat2 rebuild
PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
PREHOOK: Input: default@tbl_ice
PREHOOK: Output: default@mat2
POSTHOOK: query: alter materialized view mat2 rebuild
POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
POSTHOOK: Input: default@tbl_ice
POSTHOOK: Output: default@mat2
PREHOOK: query: select * from mat1
PREHOOK: type: QUERY
PREHOOK: Input: default@mat1
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: select * from mat1
POSTHOOK: type: QUERY
POSTHOOK: Input: default@mat1
POSTHOOK: Output: hdfs://### HDFS PATH ###
five 54
four 53
ten 60
PREHOOK: query: select * from mat2
PREHOOK: type: QUERY
PREHOOK: Input: default@mat2
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: select * from mat2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@mat2
POSTHOOK: Output: hdfs://### HDFS PATH ###
one 50
three 52
two 51
PREHOOK: query: insert into tbl_ice values (20, 'twenty', 80)
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
PREHOOK: Output: default@tbl_ice
POSTHOOK: query: insert into tbl_ice values (20, 'twenty', 80)
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@tbl_ice
PREHOOK: query: alter materialized view mat1 rebuild
PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
PREHOOK: Input: default@tbl_ice
Expand All @@ -267,3 +340,4 @@ POSTHOOK: Output: hdfs://### HDFS PATH ###
five 54
four 53
ten 60
twenty 80
18 changes: 0 additions & 18 deletions ...oop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,15 @@
import org.apache.calcite.tools.Frameworks;
import org.apache.hadoop.hive.common.TableName;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.api.LockState;

Check warning on line 36 in ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Unused import - org.apache.hadoop.hive.metastore.api.LockState. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZznoata9H46KulFlCFz&open=AZznoata9H46KulFlCFz&pullRequest=6330

Check warning on line 36 in ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Remove this unused import 'org.apache.hadoop.hive.metastore.api.LockState'. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZznoata9H46KulFlCFw&open=AZznoata9H46KulFlCFw&pullRequest=6330
import org.apache.hadoop.hive.ql.Context;
import org.apache.hadoop.hive.ql.ErrorMsg;
import org.apache.hadoop.hive.ql.QueryProperties;
import org.apache.hadoop.hive.ql.QueryState;
import org.apache.hadoop.hive.ql.ddl.DDLSemanticAnalyzerFactory.DDLType;
import org.apache.hadoop.hive.ql.io.AcidUtils;
import org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager;

Check warning on line 43 in ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Remove this unused import 'org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager'. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZznoata9H46KulFlCFx&open=AZznoata9H46KulFlCFx&pullRequest=6330

Check warning on line 43 in ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Unused import - org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZznoata9H46KulFlCF0&open=AZznoata9H46KulFlCF0&pullRequest=6330
import org.apache.hadoop.hive.ql.lockmgr.LockException;

Check warning on line 44 in ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Remove this unused import 'org.apache.hadoop.hive.ql.lockmgr.LockException'. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZznoata9H46KulFlCFy&open=AZznoata9H46KulFlCFy&pullRequest=6330

Check warning on line 44 in ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Unused import - org.apache.hadoop.hive.ql.lockmgr.LockException. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZznoata9H46KulFlCF1&open=AZznoata9H46KulFlCF1&pullRequest=6330
import org.apache.hadoop.hive.ql.log.PerfLogger;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.metadata.HiveRelOptMaterialization;
Expand Down Expand Up @@ -205,24 +205,6 @@
tableName.getEscapedNotEmptyDbTable(), viewText);
rewrittenAST = ParseUtils.parse(rewrittenInsertStatement, ctx);
this.ctx.addSubContext(ctx);

if (!this.ctx.isExplainPlan() && (AcidUtils.isTransactionalTable(table) ||
table.isNonNative() && table.getStorageHandler().areSnapshotsSupported())) {
// Acquire lock for the given materialized view. Only one rebuild per materialized view can be triggered at a
// given time, as otherwise we might produce incorrect results if incremental maintenance is triggered.
HiveTxnManager txnManager = getTxnMgr();
LockState state;
try {
state = txnManager.acquireMaterializationRebuildLock(
tableName.getDb(), tableName.getTable(), txnManager.getCurrentTxnId()).getState();
} catch (LockException e) {
throw new SemanticException("Exception acquiring lock for rebuilding the materialized view", e);
}
if (state != LockState.ACQUIRED) {
throw new SemanticException(
"Another process is rebuilding the materialized view " + tableName.getNotEmptyDbTable());
}
}
} catch (Exception e) {
throw new SemanticException(e);
}
Expand Down
Loading