Skip to content

Fix write retry, pipe fallback, and snapshot recovery edge cases#18180

Open
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:fix/write-pipe-snapshot-recovery
Open

Fix write retry, pipe fallback, and snapshot recovery edge cases#18180
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:fix/write-pipe-snapshot-recovery

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Keep tablet retries and failure reporting consistent

When an insert tablet spans multiple processors or time partitions, a type inconsistency in a later fragment used to replay already successful fragments. Track processed rows across the retry and rebuild only the remaining ranges.

Memory reservation can also fail after earlier tablet ranges or aligned/non-aligned row groups have reserved memory. Roll back those earlier reservations, preserve the original rejection status, and mark every range that was not inserted as failed so last-cache updates and consensus retry decisions do not treat missing rows as successful.

Make both V2 pipe builders lossless on fallback

PipeTableModelTsFileBuilderV2 previously did not buffer data into its fallback builder, so a primary conversion failure could return an empty successful result. Both V2 builders now also delete primary-path files when switching to fallback, including files already sealed before a later database fails.

The tree-model V2 builder now skips an individual null schema instead of dropping every valid column after it, and handles null DATE values without forcing fallback.

Replace all derived metadata during snapshot loading

Loading a template snapshot now replaces both template indexes, and an empty template-preset snapshot clears stale preset state. QuotaInfo.clear() also clears the derived space-quota usage map so entries created after a snapshot do not survive recovery.

Distinguish compatibility EOF from corrupted subscription snapshots

Commit-progress restoration uses full reads so legal short reads are accepted. A completely absent commit-progress section remains valid for older snapshots, while a partially present 1-3 byte size header now fails snapshot loading instead of silently discarding committed progress.

Verification

  • mvn spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode,iotdb-core/confignode
  • CommitProgressKeeperTest: 5 tests passed
  • QuotaInfoTest,TemplateTableTest,TemplatePreSetTableTest: 6 tests passed with -DforkCount=0
  • PipeTsFileBuilderV2Test: 4 tests passed with -DforkCount=0
  • Targeted DataRegionTest methods: 3 tests passed with -DforkCount=0
  • TsFileProcessorTest: 16 tests passed with -DforkCount=0
  • mvn -pl iotdb-core/node-commons,iotdb-core/datanode,iotdb-core/confignode -DskipTests -P with-zh-locale test-compile

This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent where it would not be obvious.
  • added unit tests or modified existing tests to cover the new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • DataRegion and TsFileProcessor
  • PipeTableModelTsFileBuilderV2 and PipeTreeModelTsFileBuilderV2
  • TemplateTable, TemplatePreSetTable, and QuotaInfo
  • CommitProgressKeeper

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant