[Bug] HMS incremental events have multiple metadata cache consistency gaps

[wenzhenghu]

Search before asking

I searched existing Apache Doris issues and did not find an issue covering these HMS incremental-event cache consistency problems.

Version

Apache Doris master at 3a75387e61388bd886eeef38b794d5ed4eb298bf.

These problems were found while reviewing #65126, but they are already present on the current master baseline and were not introduced by that PR.

What's Wrong?

There are four independent correctness gaps in HMS incremental event handling.

1. ALTER TABLE/ALTER DATABASE rename can be incorrectly cancelled by a load-through "local existence" check

AlterTableEvent.processRename() calls externalTableExistInLocal() before applying the rename:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/AlterTableEvent.java

Lines 107 to 123 in 3a75387

	private void processRename() throws DdlException {
	if (!isRename) {
	return;
	}
	boolean hasExist = Env.getCurrentEnv().getCatalogMgr()
	.externalTableExistInLocal(tableAfter.getDbName(), tableAfter.getTableName(), catalogName);
	if (hasExist) {
	logInfo("AlterExternalTable canceled,because tableAfter has exist, "
	+ "catalogName:[{}],dbName:[{}],tableName:[{}]",
	catalogName, dbName, tableAfter.getTableName());
	return;
	}
	Env.getCurrentEnv().getCatalogMgr()
	.unregisterExternalTable(tableBefore.getDbName(), tableBefore.getTableName(), catalogName, true);
	Env.getCurrentEnv().getCatalogMgr()
	.registerExternalTableFromEvent(
	tableAfter.getDbName(), tableAfter.getTableName(), catalogName, eventTime, true);

However, HMSExternalCatalog.tableExistInLocal() eventually calls the normal load-through table lookup rather than a cache-only lookup:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java

Lines 190 to 198 in 3a75387

	@Override
	public boolean tableExistInLocal(String dbName, String tblName) {
	makeSureInitialized();
	HMSExternalDatabase hmsExternalDatabase = (HMSExternalDatabase) getDbNullable(dbName);
	if (hmsExternalDatabase == null) {
	return false;
	}
	return hmsExternalDatabase.getTable(tblName).isPresent();

After HMS has renamed old_table to new_table, querying new_table from this check can load it from HMS and return true. The event is then cancelled, and cached state for old_table may remain. AlterDatabaseEvent.processRename() has the same structural problem through catalog.getDbNullable(dbAfter.getName()), although database rename is less commonly reachable in standard Hive.

2. ADD/DROP PARTITION events do not fence an in-flight partition-values load, and DROP may skip lower-level cache invalidation

Both addPartitionsCache() and dropPartitionsCache() return immediately when the partition-values entry is absent:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java

Lines 707 to 746 in 3a75387

	private void addPartitionsCache(NameMapping nameMapping,
	List<String> partitionNames,
	List<Type> partitionColumnTypes) {
	long catalogId = nameMapping.getCtlId();
	MetaCacheEntry<PartitionValueCacheKey, HivePartitionValues> partitionValuesEntry =
	partitionValuesEntryIfInitialized(catalogId);
	if (partitionValuesEntry == null) {
	return;
	}
	PartitionValueCacheKey key = new PartitionValueCacheKey(nameMapping, partitionColumnTypes);
	HivePartitionValues partitionValues = partitionValuesEntry.getIfPresent(key);
	if (partitionValues == null) {
	return;
	}
	HivePartitionValues copy = partitionValues.copy();
	Map<String, PartitionItem> nameToPartitionItem = copy.getNameToPartitionItem();
	Map<String, List<String>> nameToPartitionValues = copy.getNameToPartitionValues();
	HMSExternalCatalog catalog = hmsCatalog(catalogId);
	String localTblName = nameMapping.getLocalTblName();
	for (String partitionName : partitionNames) {
	if (nameToPartitionItem.containsKey(partitionName)) {
	LOG.info("addPartitionsCache partitionName:[{}] has exist in table:[{}]",
	partitionName, localTblName);
	continue;
	}
	ListPartitionItem listPartitionItem = toListPartitionItem(partitionName, key.types, catalog.getName());
	nameToPartitionItem.put(partitionName, listPartitionItem);
	nameToPartitionValues.put(partitionName, HiveUtil.toPartitionValues(partitionName));
	}
	copy.rebuildSortedPartitionRanges();
	HivePartitionValues partitionValuesCur = partitionValuesEntry.getIfPresent(key);
	if (partitionValuesCur == partitionValues) {
	partitionValuesEntry.put(key, copy);
	}
	}

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java

Lines 748 to 789 in 3a75387

	private void dropPartitionsCache(ExternalTable dorisTable,
	List<String> partitionNames,
	boolean invalidPartitionCache) {
	NameMapping nameMapping = dorisTable.getOrBuildNameMapping();
	long catalogId = nameMapping.getCtlId();
	MetaCacheEntry<PartitionValueCacheKey, HivePartitionValues> partitionValuesEntry =
	partitionValuesEntryIfInitialized(catalogId);
	if (partitionValuesEntry == null) {
	return;
	}
	PartitionValueCacheKey key = new PartitionValueCacheKey(nameMapping, null);
	HivePartitionValues partitionValues = partitionValuesEntry.getIfPresent(key);
	if (partitionValues == null) {
	return;
	}
	HivePartitionValues copy = partitionValues.copy();
	Map<String, PartitionItem> nameToPartitionItem = copy.getNameToPartitionItem();
	Map<String, List<String>> nameToPartitionValues = copy.getNameToPartitionValues();
	for (String partitionName : partitionNames) {
	if (!nameToPartitionItem.containsKey(partitionName)) {
	LOG.info("dropPartitionsCache partitionName:[{}] not exist in table:[{}]",
	partitionName, nameMapping.getFullLocalName());
	continue;
	}
	nameToPartitionItem.remove(partitionName);
	nameToPartitionValues.remove(partitionName);
	if (invalidPartitionCache) {
	invalidatePartitionCache(nameMapping, partitionName);
	}
	}
	copy.rebuildSortedPartitionRanges();
	HivePartitionValues partitionValuesCur = partitionValuesEntry.getIfPresent(key);
	if (partitionValuesCur == partitionValues) {
	partitionValuesEntry.put(key, copy);
	}
	}

This causes two problems:

• If a partition-values load started before the event, the event does not invalidate/bump the entry, so the pre-event snapshot may still be published afterward.
• If the partition-values entry has been evicted while partition metadata or file listings remain cached, a DROP PARTITION event returns before invalidating those lower-level entries.

The second case can expose stale data when a partition name is later recreated with a different path or files. It also prevents the cache-miss fallback added by #65334 from being reached through this DROP event path.

3. Incremental CREATE_DATABASE/CREATE_TABLE events can bypass include/exclude visibility filters

Full names loading applies include_database_list, exclude_database_list, and include_table_list:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java

Lines 551 to 616 in 3a75387

	private List<Pair<String, String>> getFilteredDatabaseNames() {
	List<String> allDatabases = Lists.newArrayList(listDatabaseNames());
	allDatabases.remove(InfoSchemaDb.DATABASE_NAME);
	allDatabases.add(InfoSchemaDb.DATABASE_NAME);
	allDatabases.remove(MysqlDb.DATABASE_NAME);
	allDatabases.add(MysqlDb.DATABASE_NAME);
	Map<String, Boolean> includeDatabaseMap = getIncludeDatabaseMap();
	Map<String, Boolean> excludeDatabaseMap = getExcludeDatabaseMap();
	lowerCaseToDatabaseName.clear();
	List<Pair<String, String>> remoteToLocalPairs = Lists.newArrayList();
	allDatabases = allDatabases.stream().filter(dbName -> {
	if (!dbName.equals(InfoSchemaDb.DATABASE_NAME) && !dbName.equals(MysqlDb.DATABASE_NAME)) {
	// Exclude database map take effect with higher priority over include database map
	if (!excludeDatabaseMap.isEmpty() && excludeDatabaseMap.containsKey(dbName)) {
	return false;
	}
	if (!includeDatabaseMap.isEmpty() && !includeDatabaseMap.containsKey(dbName)) {
	return false;
	}
	}
	return true;
	}).collect(Collectors.toList());
	for (String remoteDbName : allDatabases) {
	String localDbName = fromRemoteDatabaseName(remoteDbName);
	// Populate lowercase mapping for case-insensitive lookups
	lowerCaseToDatabaseName.put(remoteDbName.toLowerCase(), remoteDbName);
	// Apply lower_case_database_names mode to local name
	int dbNameMode = getLowerCaseDatabaseNames();
	if (dbNameMode == 1) {
	localDbName = localDbName.toLowerCase();
	} else if (dbNameMode == 2) {
	// Mode 2: preserve original remote case for display
	localDbName = remoteDbName;
	}
	remoteToLocalPairs.add(Pair.of(remoteDbName, localDbName));
	}
	// Check for conflicts when lower_case_meta_names = true or lower_case_database_names = 2
	if (Boolean.parseBoolean(getLowerCaseMetaNames()) || getLowerCaseDatabaseNames() == 2) {
	// Map to track lowercase local names and their corresponding remote names
	Map<String, List<String>> lowerCaseToRemoteNames = Maps.newHashMap();
	// Collect lowercased local names and their remote counterparts
	for (Pair<String, String> pair : remoteToLocalPairs) {
	String lowerCaseLocalName = pair.second.toLowerCase();
	lowerCaseToRemoteNames.computeIfAbsent(lowerCaseLocalName, k -> Lists.newArrayList()).add(pair.first);
	}
	// Identify conflicts: multiple remote names mapping to the same lowercase local name
	List<String> conflicts = lowerCaseToRemoteNames.values().stream()
	.filter(remoteNames -> remoteNames.size() > 1) // Conflict: more than one remote name
	.flatMap(List::stream) // Collect all conflicting remote names
	.collect(Collectors.toList());
	// Throw exception if conflicts are found
	if (!conflicts.isEmpty()) {
	throw new RuntimeException(String.format(
	FOUND_CONFLICTING + " database names under case-insensitive conditions. "
	+ "Conflicting remote database names: %s in catalog %s. "
	+ "Please use meta_names_mapping to handle name mapping.",
	String.join(", ", conflicts), name));
	}

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java

Lines 343 to 356 in 3a75387

	* @return names of tables in specified database, filtered by include_table_list if configured
	*/
	public final List<String> listTableNames(SessionContext ctx, String dbName) {
	makeSureInitialized();
	Map<String, List<String>> includeTableMap = getIncludeTableMap();
	if (includeTableMap.containsKey(dbName) && !includeTableMap.get(dbName).isEmpty()) {
	if (LOG.isDebugEnabled()) {
	LOG.debug("get table list from include map. catalog: {}, db: {}, tables: {}",
	name, dbName, includeTableMap.get(dbName));
	}
	return includeTableMap.get(dbName);
	}
	return listTableNamesFromRemote(ctx, dbName);
	}

The incremental register paths directly update the legacy metadata cache without applying the same visibility policy:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java

Lines 205 to 216 in 3a75387

	@Override
	public void registerDatabase(long dbId, String dbName) {
	if (LOG.isDebugEnabled()) {
	LOG.debug("create database [{}]", dbName);
	}
	ExternalDatabase<? extends ExternalTable> db = buildDbForInit(dbName, null, dbId, logType, false);
	if (isInitialized()) {
	metaCache.updateCache(db.getRemoteName(), db.getFullName(), db,
	Util.genIdByName(name, db.getFullName()));
	}

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java

Lines 649 to 663 in 3a75387

	// Only used for sync hive metastore event
	@Override
	public boolean registerTable(TableIf tableIf) {
	makeSureInitialized();
	String tableName = tableIf.getName();
	if (LOG.isDebugEnabled()) {
	LOG.debug("create table [{}]", tableName);
	}
	if (isInitialized()) {
	String localName = extCatalog.fromRemoteTableName(this.remoteName, tableName);
	metaCache.updateCache(tableName, localName, (T) tableIf,
	Util.genIdByName(extCatalog.getName(), name, localName));
	lowerCaseToTableName.put(tableName.toLowerCase(), tableName);
	}
	setLastUpdateTime(System.currentTimeMillis());

When a names cache is already hot, an HMS CREATE event can therefore add a database/table that should remain hidden. The object can subsequently become queryable through the cached name.

4. DROP_PARTITION records a DATABASE deletion in ExternalMetaIdMgr

DropPartitionEvent.transferToMetaIdMappings() uses META_OBJECT_TYPE_DATABASE instead of META_OBJECT_TYPE_PARTITION:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/DropPartitionEvent.java

Lines 142 to 152 in 3a75387

	@Override
	protected List<MetaIdMappingsLog.MetaIdMapping> transferToMetaIdMappings() {
	List<MetaIdMappingsLog.MetaIdMapping> metaIdMappings = Lists.newArrayList();
	for (String partitionName : this.getAllPartitionNames()) {
	MetaIdMappingsLog.MetaIdMapping metaIdMapping = new MetaIdMappingsLog.MetaIdMapping(
	MetaIdMappingsLog.OPERATION_TYPE_DELETE,
	MetaIdMappingsLog.META_OBJECT_TYPE_DATABASE,
	dbName, tblName, partitionName);
	metaIdMappings.add(metaIdMapping);
	}
	return ImmutableList.copyOf(metaIdMappings);

ExternalMetaIdMgr interprets this as a request to remove the entire database mapping subtree:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java

Lines 145 to 169 in 3a75387

	private static void handleDelMetaIdMapping(MetaIdMappingsLog.MetaIdMapping mapping,
	CtlMetaIdMgr ctlMetaIdMgr,
	MetaIdMappingsLog.MetaObjectType objType) {
	TblMetaIdMgr tblMetaIdMgr;
	DbMetaIdMgr dbMetaIdMgr;
	switch (objType) {
	case DATABASE:
	ctlMetaIdMgr.dbNameToMgr.remove(mapping.getDbName());
	break;
	case TABLE:
	dbMetaIdMgr = ctlMetaIdMgr.dbNameToMgr.get(mapping.getDbName());
	if (dbMetaIdMgr != null) {
	dbMetaIdMgr.tblNameToMgr.remove(mapping.getTblName());
	}
	break;
	case PARTITION:
	dbMetaIdMgr = ctlMetaIdMgr.dbNameToMgr.get(mapping.getDbName());
	if (dbMetaIdMgr != null) {
	tblMetaIdMgr = dbMetaIdMgr.tblNameToMgr.get(mapping.getTblName());
	if (tblMetaIdMgr != null) {
	tblMetaIdMgr.partitionNameToMgr.remove(mapping.getPartitionName());
	}
	}

Current production use of the ExternalMetaIdMgr lookup APIs is limited, so the immediate query impact appears limited, but the persisted/replayed external metadata ID state is incorrect.

What You Expected?

• Rename events should inspect only locally published cache state and should always remove the old local name.
• Partition events should prevent pre-event snapshots from being published and should invalidate partition/file caches even when partition-values cache state is cold.
• Incremental create events should apply the same visibility rules as full names loading.
• DROP_PARTITION should delete only the corresponding partition metadata-ID mapping.

How to Reproduce?

Rename

1. Enable HMS incremental event synchronization.
2. Ensure the target name is not locally cached.
3. Rename old_table to new_table in Hive.
4. Process the ALTER_TABLE event.
5. Observe that the existence check loads new_table from HMS and cancels the event; cached old_table state may remain.

Partition cache

1. Cache partition metadata/file listings for p=1.
2. Evict or invalidate only the table's partition-values entry, or block an in-flight partition-values load.
3. Drop p=1 through HMS and process the DROP_PARTITION event.
4. Recreate p=1 with a different location/files.
5. The old partition metadata or file listing may still be reused.

Visibility filter

1. Create an HMS catalog with an include/exclude database filter or include_table_list.
2. Warm the corresponding names cache.
3. Create a filtered-out database/table directly in Hive.
4. Process the HMS CREATE event.
5. Observe that the filtered object is inserted into the cached names and becomes visible/queryable.

External metadata ID mapping

1. Add a database/table/partition mapping to ExternalMetaIdMgr.
2. Process the partition's DROP_PARTITION mapping.
3. Observe that the database mapping subtree is removed instead of only the partition mapping.

Suggested Fixes

1. Introduce a cache-only local-existence check for rename handling. Always invalidate the old name; only skip publishing the new object when it is already locally present.
2. For ADD/DROP PARTITION, invalidate or bump the partition-values key when it is cold/loading. DROP should unconditionally invalidate partition metadata/file cache entries for each dropped partition before optionally updating a hot partition-values snapshot.
3. Reuse a shared database/table visibility predicate in both full names loading and incremental registration.
4. Change the DROP_PARTITION mapping type to META_OBJECT_TYPE_PARTITION and add an event mapping unit test.

The four fixes can be implemented as separate PRs if preferred.

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_8fa76c90e7f5
This content is generated by AI for reference only.

Initial triage: I re-read the live issue and checked the referenced code at 3a75387e61388bd886eeef38b794d5ed4eb298bf. I also refreshed upstream/master to 9a5afd4958f2794a2dd458b2e7463315e9746ce3; the relevant files have no diff from the issue commit, so these observations still apply to the current master snapshot I inspected.

Judgment:

• The report looks valid and is best handled as four independent FE HMS incremental-event cache consistency bugs.
• I did not find evidence in the code path that makes any of the four cases intentionally safe.

Code-backed details:

1. Rename cancellation path: confirmed. AlterTableEvent.processRename() calls externalTableExistInLocal(tableAfter...); for HMS, this reaches HMSExternalCatalog.tableExistInLocal(), which calls getDbNullable() and hmsExternalDatabase.getTable(tblName). ExternalCatalog.getDbNullable() and ExternalDatabase.getTableNullable() both use metaCache.getMetaObj(...), and MetaCache.getMetaObj() loads on miss. That makes the check not cache-only. If HMS already exposes the renamed target, the check can materialize the new table and return true; processRename() then returns before unregistering tableBefore, so the old local name can remain. AlterDatabaseEvent.processRename() has the analogous risk because catalog.getDbNullable(dbAfter.getName()) is also load-through; getDbForReplay(...) shows there is already a cache-only pattern available for databases.

2. Partition events: confirmed, with a concurrency nuance. MetaCacheEntry has generation-based protection against stale in-flight manual loads, but only when an invalidation actually happens. In HiveExternalMetaCache.PartitionCacheCoordinator.addPartitionsCache() and dropPartitionsCache(), the code returns when the partition-values entry/key is not present, so no generation is bumped for an in-flight partition-values load. In dropPartitionsCache(), this early return also prevents invalidatePartitionCache(...) from running, so stale partition metadata and file-listing entries can survive after a DROP. The lower-level invalidation helper can already fall back to table-id + partition-values matching when the exact partition metadata is cold, but the current DROP event path can skip calling that helper entirely.

3. Visibility filters: confirmed. Full names loading applies include_database_list, exclude_database_list, and include_table_list through ExternalCatalog.getFilteredDatabaseNames() and ExternalCatalog.listTableNames(). Incremental CREATE paths bypass those predicates: CreateDatabaseEvent -> CatalogMgr.registerExternalDatabaseFromEvent() -> HMSExternalCatalog.registerDatabase() updates the catalog meta cache directly, and CreateTableEvent -> CatalogMgr.registerExternalTableFromEvent() -> ExternalDatabase.registerTable() updates the DB meta cache and lowerCaseToTableName directly. If the names cache is hot, a filtered object can be inserted into the visible cached names.

4. ExternalMetaIdMgr mapping: confirmed. DropPartitionEvent.transferToMetaIdMappings() emits META_OBJECT_TYPE_DATABASE for a partition delete. ExternalMetaIdMgr.handleDelMetaIdMapping() interprets DATABASE delete by removing the whole dbNameToMgr entry, while PARTITION delete would only remove partitionNameToMgr under the table. This is a direct type mismatch.

Missing information / validation needed:

• For (1), a small HMS rename regression test should assert that the target-name existence probe is cache-only and that the old local table/db name is always removed.
• For (2), a regression/unit test should cover a DROP with a hot lower-level partition/file cache but a cold partition-values cache, plus an in-flight partition-values load where the event occurs before the load publishes.
• For (3), tests should cover both database and table include/exclude filters with a pre-warmed names cache before processing CREATE_DATABASE / CREATE_TABLE events.
• For (4), an event mapping replay test should verify DROP_PARTITION deletes only the partition mapping and preserves sibling table/db mappings.

Suggested next steps:

• Split into separate PRs if that is easier to review: rename cache-only existence, partition event fencing/invalidation, shared visibility predicates for incremental register paths, and the DROP_PARTITION meta-id type fix.
• Prefer sharing the same database/table visibility helper between full-list loading and event registration, so incremental events cannot drift from catalog-list semantics.
• For partition events, make DROP invalidate partition metadata/file cache entries before attempting the hot partition-values snapshot update, and make ADD/DROP invalidate or generation-bump the partition-values key when the entry exists but the value is cold/loading.
• For rename, use cache-only lookup APIs similar to getDbForReplay / getTableForReplay for local-existence checks, and avoid returning before removing the old name.