Skip to content

Extend region migration speed limit to transfers and deletion#18118

Open
CRZbulabula wants to merge 2 commits into
masterfrom
feat-region-delete-speed-limit
Open

Extend region migration speed limit to transfers and deletion#18118
CRZbulabula wants to merge 2 commits into
masterfrom
feat-region-delete-speed-limit

Conversation

@CRZbulabula

@CRZbulabula CRZbulabula commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Extend region_migration_speed_limit_bytes_per_second to throttle region migration transfer across consensus protocols that have real transfer paths.
  • Keep IoTConsensus AddPeer snapshot transfer covered by the shared limiter, and add coverage for IoTConsensusV2 consensus pipe transfer and Ratis install snapshot transfer.
  • Share the same limiter with region file deletion and wire DataRegion, SchemaRegion, WAL, snapshot, and consensus peer cleanup paths through rate-limited deletion helpers.
  • Update the config template description and add unit coverage for rate-limited deletion, IoTConsensusV2 transfer size calculation, and Ratis snapshot chunk size calculation.

Protocol coverage

  • IoTConsensus: AddPeer snapshot transfer is rate-limited.
  • IoTConsensusV2: tablet, tablet batch, delete event, and TsFile piece transfer requests are rate-limited.
  • Ratis: install snapshot file chunks are rate-limited before sending.
  • SimpleConsensus: no remote region migration transfer path; local region deletion cleanup remains rate-limited.

Tests

  • mvn spotless:apply -pl iotdb-core/node-commons,iotdb-core/consensus,iotdb-core/datanode
  • mvn test -pl iotdb-core/node-commons -am -Dtest=FileUtilsTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
  • mvn test -pl iotdb-core/consensus -am -Dtest=RateLimitedGrpcLogAppenderTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
  • mvn test -pl iotdb-core/datanode -am -Dtest=IoTConsensusV2RateLimiterTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
  • mvn compile -pl iotdb-core/consensus,iotdb-core/datanode -am -DskipTests
  • mvn test-compile -DskipTests -P with-zh-locale -pl iotdb-core/consensus,iotdb-core/datanode -am

Note: full reactor mvn test-compile -DskipTests reaches the distribution module and fails to resolve existing 2.0.11-SNAPSHOT zip artifacts for iotdb-server, iotdb-cli, and iotdb-confignode. The changed modules compile successfully.

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.55932% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.73%. Comparing base (8940ba0) to head (801d4fb).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
...rotocol/iotconsensusv2/IoTConsensusV2SyncSink.java 0.00% 14 Missing ⚠️
...ocol/iotconsensusv2/IoTConsensusV2RateLimiter.java 69.23% 4 Missing ⚠️
...b/schemaengine/schemaregion/SchemaRegionUtils.java 0.00% 4 Missing ⚠️
...java/org/apache/iotdb/commons/utils/FileUtils.java 71.42% 4 Missing ⚠️
...otdb/commons/utils/RegionMigrationRateLimiter.java 81.25% 3 Missing ⚠️
...rg/apache/iotdb/consensus/pipe/IoTConsensusV2.java 0.00% 2 Missing ⚠️
...gine/schemaregion/impl/SchemaRegionMemoryImpl.java 0.00% 2 Missing ⚠️
...gine/schemaregion/impl/SchemaRegionPBTreeImpl.java 0.00% 2 Missing ⚠️
...g/apache/iotdb/db/storageengine/StorageEngine.java 0.00% 2 Missing ⚠️
.../iotdb/db/storageengine/dataregion/DataRegion.java 66.66% 2 Missing ⚠️
... and 4 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18118      +/-   ##
============================================
- Coverage     41.84%   41.73%   -0.11%     
  Complexity      318      318              
============================================
  Files          5296     5301       +5     
  Lines        373593   373684      +91     
  Branches      48234    48234              
============================================
- Hits         156313   155960     -353     
- Misses       217280   217724     +444     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CRZbulabula CRZbulabula changed the title Limit region deletion with migration speed limit Extend region migration speed limit to transfers and deletion Jul 6, 2026
@sonarqubecloud

sonarqubecloud Bot commented Jul 6, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant