Cyb/metadata table by alpass163gmail · Pull Request #18127 · apache/iotdb

alpass163gmail · 2026-07-06T10:37:57Z

This PR introduces a lease-based self-fencing framework to enable high
availability for table metadata operations. It prevents DataNodes from
serving stale schema during network partitions.

Problem

Currently, in a cluster deployment (e.g., 1 ConfigNode, 3 DataNodes),
metadata operations lack true high availability. If a single DataNode (DN)
crashes or experiences a network partition, execution of DDL procedures
(such as Create/Alter/Drop Table, View management, TTL adjustments, and
Drop Database) will directly fail and trigger a rollback.

Solution

Lease Mechanism

MetadataLeaseManager (DataNode): Tracks lease via ConfigNode heartbeats.
Uses monotonic clock. Self-fences (clears cache, blocks reads/writes) if no
heartbeat received within metadata_lease_fence_ms (T_fence).
MetadataLeaseFencedException: Thrown when operations are blocked on a
fenced DataNode.

Broadcast Coordination

DataNodeContactTracker (ConfigNode): Records time of last successful
heartbeat response per DataNode. Separately maintained from load-balancing
samples to ensure correctness.
MetadataBroadcastVerdict: Pure decision logic — PROCEED if all unacked
DataNodes have been silent for T_proceed = T_fence + margin, WAIT
otherwise, FAIL when retry budget exhausted.
ClusterCachePropagator: Broadcasts cache invalidations with retry loop,
waiting up to T_proceed for unresponsive DataNodes to prove self-fenced.

Schema Change Integration

Procedures now propagate metadata invalidations via
ClusterCachePropagator before proceeding.
Pre-deletion marker (PreDeleteTsTable) added for safe table state
transitions.
Rollback mechanism (RollbackPreDeleteTablePlan) for failed schema
changes.

Configuration

New config: metadata_lease_fence_ms (default in
iotdb-system.properties.template).

Testing

Unit tests for MetadataLeaseManager, DataNodeContactTracker,
MetadataBroadcastVerdict, ClusterCachePropagator
Lease integration tests for DataNodeTableCache, PartitionCache,
ClusterAuthorityFetcher
New HA IT: IoTDBTableDDLHAIT

Key Components Added

MetadataLeaseManager (DataNode): Tracks the lease and performs
self-fencing upon expiration.
DataNodeContactTracker (ConfigNode): Tracks the last successful heartbeat
timestamp for each DataNode.
ClusterCachePropagator (ConfigNode): Broadcasts cache invalidations with
a fencing-aware retry mechanism.
MetadataBroadcastVerdict (ConfigNode): Decides when unacknowledged
DataNodes are logically safe to skip.
MetadataLeaseFencedException (node-commons): The explicit exception
thrown on a fenced DataNode.
PreDeleteTsTable (node-commons): Represents the marker table state to
ensure safe schema deletion.

…lity of table metadata operation procedure

… lease frame

…ager, add the scheduled task to check the status of metadata

# Conflicts: # iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ProcedureMessages.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/env/ConfigNodeProcedureEnv.java # iotdb-core/datanode/src/main/i18n/zh/org/apache/iotdb/db/i18n/DataNodePipeMessages.java # iotdb-core/datanode/src/main/i18n/zh/org/apache/iotdb/db/i18n/DataNodeSchemaMessages.java

…m CommonConfig to the IoTDBConfig

JackieTien97 · 2026-07-07T06:19:59Z

          MetricService.getInstance(),
          NodeUrlUtils.convertTEndPointUrl(dataNodeLocation.getClientRpcEndPoint()));
+      // Drop the removed DataNode's metadata-lease contact/capability state so it is not retained,
+      // and a future DataNode reusing the id cannot inherit stale fencing history.


id will always increase, no new DN will reuse removed DN's id

JackieTien97 · 2026-07-07T06:20:36Z

  }

+  // only care about the AbstractAlterOrDropTableProcedure(except the drop table/view)
+  // and the DeleteDatabaseProcedure


explain why we need to exclude the drop table/view procedure

JackieTien97 · 2026-07-07T06:20:59Z

+    // DDL: a fenced DataNode fails closed on its now-stale table cache and resyncs on lease
+    // recovery, so it cannot serve dirty schema. Only fail if an unacked DataNode is not provably
+    // fenced (it may still be serving clients).
+    final TUpdateTableReq req = SchemaUtils.BuildPreUpdateTableReq(database, table, oldName);


Suggested change

final TUpdateTableReq req = SchemaUtils.BuildPreUpdateTableReq(database, table, oldName);

final TUpdateTableReq req = SchemaUtils.buildPreUpdateTableReq(database, table, oldName);

JackieTien97 · 2026-07-07T06:21:40Z

          database,
          table.getTableName(),
-          failedResults);
+          "an unreachable DataNode is not provably fenced");


follow the i18n standarn, define a constant in both zh and en.

JackieTien97 · 2026-07-07T06:22:01Z

          database,
          table.getTableName(),
-          failedResults);
+          "an unreachable DataNode is not provably fenced");


JackieTien97 · 2026-07-07T06:24:03Z

-                    ProcedureMessages.INVALIDATE_VIEW_SCHEMAENGINE_CACHE_FAILED)));
-        return;
-      }
+      LOGGER.error(


JackieTien97 · 2026-07-07T06:25:40Z

-                new MetadataException(ProcedureMessages.INVALIDATE_SCHEMAENGINE_CACHE_FAILED)));
-        return;
-      }
+      LOGGER.error(


JackieTien97 · 2026-07-07T06:25:50Z

-                    ProcedureMessages.INVALIDATE_VIEW_SCHEMAENGINE_CACHE_FAILED)));
-        return;
-      }
+      LOGGER.error(


JackieTien97 · 2026-07-07T06:25:57Z

-                new MetadataException(ProcedureMessages.INVALIDATE_SCHEMAENGINE_CACHE_FAILED)));
-        return;
-      }
+      LOGGER.error(


JackieTien97 · 2026-07-07T06:44:46Z

+import java.io.OutputStream;
+
+public class PreDeleteTsTable extends TsTable {
+  public static final int PRE_DELETE_MARKER = -2;


Suggested change

public static final int PRE_DELETE_MARKER = -2;

public static final int PRE_DELETE_MARKER = -3;

put it as an enum, add comments that -2 is reserved and don't use.

Caideyipi · 2026-07-07T07:54:51Z

I reviewed this PR against the HA requirement/design docs. The direction looks good, but I found several issues that should be addressed before merge:

The DataNode fencing decision is not checked synchronously on access.

MetadataLeaseManager.isFenced() only checks whether metadataState != NORMAL. Lease expiration is moved from NORMAL to NEED_CLEAR only by the scheduled checkLeaseStatus() task. If check_dn_lease_status_interval_ms is configured larger than the ConfigNode-side T_fence + margin, the ConfigNode may proceed with a metadata change while the partitioned DataNode is still in NORMAL and can keep serving stale cache. Please make isFenced() / failIfMetadataLeaseFenced() check the elapsed heartbeat age and CAS the state to NEED_CLEAR immediately when the lease is expired.

A reachable but already-fenced DataNode can make the HA broadcast fail.

DataNodeInternalRPCServiceImpl.updateTable() calls DataNodeTableCache.preUpdateTable(), which throws MetadataLeaseFencedException while the DataNode is fenced. On the ConfigNode side, ClusterCachePropagator treats any response other than SUCCESS_STATUS and CAN_NOT_CONNECT_DATANODE as FAIL. This means a DataNode that has correctly self-fenced but is still reachable, or is recovering in NEED_CLEAR / PULLING, can fail the DDL instead of being considered safe. Please return/handle METADATA_LEASE_FENCED explicitly and let the propagator treat it as a safe fenced response.

The advertised supportsFencing compatibility guard is missing.

The design says the ConfigNode should track whether each DataNode supports self-fencing, with default false for old nodes. Current ClusterCachePropagator only uses heartbeat age. During rolling upgrade or mixed-version operation, an old DataNode that does not self-fence can still be considered safe after T_proceed, which is unsafe. Please add capability reporting/tracking and only skip unreachable DataNodes that are known to support fencing.

The fetchTables Thrift change is not backward compatible.

fetchTables(map<string, set<string>> fetchTableMap) was changed to fetchTables(map<string, set<string>> fetchTableMap, byte tableNodeStatus). This breaks old DataNode/new ConfigNode compatibility: an old caller will not set the new argument, and the server can parse it as the default byte value 0 (PRE_CREATE), which then goes into the unsupported branch in ConfigManager.fetchTables(). Please use a new RPC or an optional/defaulted argument that preserves the old behavior as USING.

There are new raw English log/message literals in main code.

For example, ConfigNodeRPCServiceProcessor.reloadCacheAfterLeaseRecovery() logs Execute getMetaDataCache with result {} directly. This repo requires user/operator visible strings to go through the i18n message constants. Please move the new literals to the proper message classes.

Overall, the main concept is sound, but the above issues affect the safety and compatibility of the lease-based HA mechanism.

Yaobin Chen added 6 commits July 3, 2026 09:36

introduce the lease and self-fence mechanism to support high availabi…

d8d2591

…lity of table metadata operation procedure

set the MetadataLeaseFenceMs in UT envSetUp, remove the effect of the…

1ad5281

… lease frame

fix the problems, main modification is the change of MetadataLeaseMan…

5b26f9b

…ager, add the scheduled task to check the status of metadata

fix i18n

8b604cf

fix: remove the configuration 'check_dn_lease_status_interval_ms' fro…

2dd3759

…m CommonConfig to the IoTDBConfig

JackieTien97 requested changes Jul 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Cyb/metadata table#18127

Cyb/metadata table#18127
alpass163gmail wants to merge 6 commits into
apache:masterfrom
alpass163gmail:cyb/metadata-table

alpass163gmail commented Jul 6, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

JackieTien97 Jul 7, 2026

Uh oh!

Caideyipi commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	final TUpdateTableReq req = SchemaUtils.BuildPreUpdateTableReq(database, table, oldName);
	final TUpdateTableReq req = SchemaUtils.buildPreUpdateTableReq(database, table, oldName);

	public static final int PRE_DELETE_MARKER = -2;
	public static final int PRE_DELETE_MARKER = -3;

Uh oh!

Conversation

alpass163gmail commented Jul 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Caideyipi commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants