feat(writer): add hoodie.meta.fields.mode for selective meta-field population on CoW tables by nsivabalan · Pull Request #19205 · apache/hudi

nsivabalan · 2026-07-06T19:09:49Z

Describe the issue this Pull Request addresses

Today the only way to opt out of populating Hudi's five meta columns is the all-or-nothing hoodie.populate.meta.fields=false. That saves storage but disables incremental queries (which require _hoodie_commit_time) and file-level pruning / investigation lookups (which need _hoodie_file_name).

A community user surfaced this trade-off (#18383, also discussed at #17959). This PR proposes a scoped, list-based property that lets a table opt into _hoodie_commit_time and/or _hoodie_file_name without paying for the remaining three meta columns.

This PR supersedes #19047 (the interim hoodie.meta.fields.commit.time.enabled boolean approach). Closes #18383.

Summary and Changelog

Adds a new table property hoodie.meta.fields.mode — a comma-separated list of meta columns to populate when hoodie.populate.meta.fields=false. Allowed tokens are _hoodie_commit_time and _hoodie_file_name; any other token is rejected up-front. The mode is immutable at runtime (settable only at table creation, via hudi-cli, or during upgrade).

Resulting modes:

`populate.meta.fields`	`meta.fields.mode`	Effective mode
`true` (default)	ignored	ALL — today's default
`false`	empty	NONE — today's minimal-meta-fields
`false`	`_hoodie_commit_time`	COMMIT_TIME_ONLY
`false`	`_hoodie_file_name`	FILE_NAME_ONLY
`false`	`_hoodie_commit_time,_hoodie_file_name`	COMMIT_TIME_AND_FILE_NAME
`true`	non-empty	rejected at writer init (ambiguous)

Why a list instead of a boolean or enum

Extensible without another config flag — adding a third opt-in field later means changing the allowed set, not adding a new property.
Empty default preserves bit-identical backward compat — every existing table on disk resolves to ALL or NONE without any new property being read.
Pre-1.3.0 readers see `populate.meta.fields=false` and treat the table as NONE — they cannot do incremental queries on the table, but they don't produce silent wrong results either.
Structurally encodes "additive on NONE" — most code paths that branch on `populate.meta.fields` keep working unchanged; only paths that specifically need commit_time / file_name consult the new accessors.

Plug points

Config + accessors (`hudi-common` / `hudi-client-common`):

New `HoodieTableConfig.META_FIELDS_MODE` property + static `parseMetaFieldsMode(String)` helper.
New predicates: `isCommitTimePopulated()`, `isFileNamePopulated()`, `isRecordKeyPopulated()`.
`HoodieWriteConfig` pass-through accessors + `Builder.withMetaFieldsMode(String)`.
`HoodieWriteConfig.validate()` rejects:
- unknown token in the mode list (parser throws),
- `populate=true` + non-empty mode (ambiguous),
- `MERGE_ON_READ` + non-empty mode (log-write path not yet wired; follow-up),
- non-Spark engine + non-empty mode (Flink RowData / Java-client not yet wired; follow-up).
`HoodieTableMetaClient.TableBuilder.setMetaFieldsMode(String)` persists the mode on `hoodie.properties` at table init.
`HoodieSparkSqlWriter` wires both fresh-table and bootstrap creation paths.

Writer engines:

`HoodieAvroParquetWriter`, `HoodieSparkParquetWriter`, `HoodieRowCreateHandle` each take a `Set metaFieldsMode` and populate columns selectively. Bloom filter / record-key index registration is intentionally skipped when the record-key column is not populated.

Read path (incremental query rejection):

`IncrementalRelationV1/V2` (CoW) now check `isCommitTimePopulated()` rather than `populateMetaFields()` — COMMIT_TIME_ONLY tables are accepted, NONE tables remain rejected with a clearer message.
`MergeOnReadIncrementalRelationV1/V2` (MoR) keep the strict `populateMetaFields()` guard until the MoR log-write path is updated. Selective modes are CoW-only in this release; MoR support is a follow-up.

Runtime immutability guard (`HoodieWriterUtils.validateTableConfig`):

Explicit null-on-disk → non-empty-in-write-options transition on the mode property is now blocked with a message directing the user to CLI / upgrade. The existing loop only flagged the mismatch when the on-disk value was non-null, which let a silent-drop path slip through for older tables.

Scope

✅ Spark Avro / Spark Row writer paths.
✅ Spark Parquet bulk-insert.
✅ CoW incremental query rejection logic across V1 / V2.
❌ MoR — `HoodieAppendHandle.populateMetadataFields` is not yet updated for the new mode, so MoR + non-empty mode is rejected at writer init. Tracked as a follow-up PR.
❌ Flink RowData / Java-client writers — out of scope for this patch; the writer init rejects non-empty mode with a clear message. Tracked as a follow-up PR.
❌ ORC / HFile writers — ORC continues to populate all meta fields unconditionally (legacy behavior); HFile is used only by MDT which is always ALL.

Impact

Storage layout: no change for tables that don't opt in. New optional mode for tables that do. Default behavior unchanged.
API: no public API breakage. New table property, new accessors, new builder method — all additive.
Configuration:
- `hoodie.meta.fields.mode` (default `""`). Only meaningful when `hoodie.populate.meta.fields=false`. Persisted on `hoodie.properties` at table init. Immutable at runtime.
Performance: writer hot path adds one Set membership check per row per opt-in field. The set is parsed once in the writer constructor.
Forward-compat: pre-1.3.0 readers ignore the new property and treat the table as NONE — no silent wrong results.

Risk Level

low

Additive change with a narrow scope. The default path is untouched. Fail-fast validation at writer init rejects every ambiguous combination loudly. Runtime property changes are blocked.

Documentation Update

New config `hoodie.meta.fields.mode` documented via `@ConfigProperty` annotation on `HoodieTableConfig`.
If the website page on meta fields exists, a separate docs PR will add the new mode table.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

🤖 Generated with Claude Code

nsivabalan

Can we confirm that we can update meta fields mode value on the fly during writes.
and do we have tests across HoodieStreamer, spark datasource writes towards this.

nsivabalan · 2026-07-06T19:11:55Z

+      if (populateCommitTime) {
+        metaFields[HoodieRecord.COMMIT_TIME_METADATA_FIELD_ORD] = shouldPreserveHoodieMetadata
+            ? row.getUTF8String(HoodieRecord.COMMIT_TIME_METADATA_FIELD_ORD) : commitTime;
+        metaFields[HoodieRecord.COMMIT_SEQNO_METADATA_FIELD_ORD] = shouldPreserveHoodieMetadata


why did we include COMMIT_SEQNO_METADATA_FIELD_ORD also here?
we wanted only the commit time meta field.. since thats whats is required for incremental queries.

Addressed in cfd1fea706ed: dropped _hoodie_commit_seqno population from the selective path. Only _hoodie_commit_time is populated. Downstream consumers that assume seqno is populated whenever commit_time is (if any exist) are out of scope for this PR — the seqno column simply stays null in COMMIT_TIME_ONLY / COMMIT_TIME_AND_FILE_NAME. Rebased on latest upstream/master.

nsivabalan · 2026-07-06T19:12:41Z

+    } else if (populateCommitTime || populateFileName) {
+      if (populateCommitTime) {
+        row.update(COMMIT_TIME_METADATA_FIELD.ordinal(), instantTime);
+        row.update(COMMIT_SEQNO_METADATA_FIELD.ordinal(),


same comment as above.

Addressed in cfd1fea706ed — same as above. Seqno population dropped from the selective path in HoodieSparkParquetWriter#writeRowWithMetadata.

nsivabalan · 2026-07-06T19:17:00Z

@@ -89,7 +89,7 @@ protected HoodieFileWriter newParquetFileWriter(
        hoodieConfig.getLongOrDefault(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE),
        storageConfiguration, hoodieConfig.getDoubleOrDefault(HoodieStorageConfig.PARQUET_COMPRESSION_RATIO_FRACTION),
        hoodieConfig.getBooleanOrDefault(HoodieStorageConfig.PARQUET_DICTIONARY_ENABLED));
-    return new HoodieAvroParquetWriter(path, parquetConfig, instantTime, taskContextSupplier, populateMetaFields);
+    return new HoodieAvroParquetWriter(path, parquetConfig, instantTime, taskContextSupplier, populateMetaFields, metaFieldsMode);


can we not fold both populateMetaFields arg and metaFieldsMode into one argument. do we really need both?
not just here. in all writer factory and other places, where we are adding metaFieldsMode as new arg in addition to populateMetaFields.

Addressed in cfd1fea706ed: folded populateMetaFields + metaFieldsMode into a single MetaFieldsMode enum (ALL / NONE / COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME). The enum is now the sole meta-field argument passed through every writer factory and writer constructor. populate.meta.fields remains on the config surface for backward compat and is auto-derived from the enum. The predicate helpers isCommitTimePopulated() / isFileNamePopulated() on the enum are used in write branches. Also note: the follow-up CLI PR (#19206) uses the same enum for the table set-meta-fields-mode command surface.

nsivabalan · 2026-07-06T19:17:26Z

+      // population is intentionally skipped — that requires the record-key column.
+      GenericRecord genericRecord = (GenericRecord) avroRecord;
+      if (populateCommitTime) {
+        String seqId = HoodieRecord.generateSequenceId(instantTime,


we don't need to add commit seq no

Addressed in cfd1fea706ed — same as above. Seqno population dropped from the selective path in HoodieAvroParquetWriter#writeAvroWithMetadata.

nsivabalan · 2026-07-06T19:19:16Z

-    throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
+  if (!metaClient.getTableConfig.isCommitTimePopulated()) {
+    throw new HoodieException("Incremental queries are not supported when _hoodie_commit_time is not populated. "
+      + "Either keep hoodie.populate.meta.fields=true or include _hoodie_commit_time in hoodie.meta.fields.mode.")


lets also call out that, a table once disabled for commit time, can never be switched back. So, users got to re-create a new table.
just setting these writer props during next write may not suffice.

lets capture this in the msg.

Addressed in cfd1fea706ed: extended the error message in IncrementalRelationV1 / IncrementalRelationV2 to state that hoodie.meta.fields.mode is a physical-storage decision baked into files at write time — it cannot be flipped by changing writer options later, and a table that was created without _hoodie_commit_time populated must be recreated to enable incremental queries. The runtime-immutability guard in HoodieWriterUtils.validateTableConfig blocks silent flips (both null → non-empty and non-empty → different) with a message pointing at the CLI / upgrade path. The follow-up CLI PR (#19206) adds table set-meta-fields-mode as the sanctioned mutation path with a --force guard for tables that already have commits.

…enum Addresses review feedback on apache#19205: - Fold (populateMetaFields boolean, comma-separated mode) into a single MetaFieldsMode enum (ALL / NONE / COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME). The enum is the sole argument passed through every writer factory and writer constructor. populate.meta.fields stays on the config surface for backward compat and is auto-derived from the enum. - On-disk representation is now the enum name (e.g. "COMMIT_TIME_ONLY") instead of a comma-separated token list. Existing tables without the property continue to fall back to ALL or NONE based on the legacy hoodie.populate.meta.fields boolean. - Drop _hoodie_commit_seqno population from the selective write path — only _hoodie_commit_time is needed for incremental queries. (Review comments on HoodieRowCreateHandle:193, HoodieSparkParquetWriter:100, HoodieAvroParquetWriter:103.) - Extend the IncrementalRelation error message to state that hoodie.meta.fields.mode is physical-storage and cannot be flipped by changing write options — existing tables must be recreated to change it. - StreamSync.initializeEmptyTable() was missing the mode pass-through — wired via setMetaFieldsModeFromString. All other non-test initTable call sites either use fromProperties() (which now routes the mode through) or intentionally don't touch meta fields (CLI table create, MDT, sample writes). Test coverage additions: - TestMetaFieldsMode (hudi-spark): end-to-end assertions read parquet back and verify which meta columns are populated / null per mode. Covers row-writer path (bulk_insert with row.writer.enable=true, default) AND non-row-writer path (insert with row.writer.enable=false). Adds inline-clustering coverage for all 5 modes verifying that clustered files preserve the mode's column population semantics. - TestHoodieStreamerMetaFieldsMode (hudi-utilities): parameterized end-to-end test through the HoodieStreamer entrypoint covering all 5 modes plus MoR+selective rejection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…pulation on CoW tables Introduces `hoodie.meta.fields.mode` — a comma-separated list of meta columns to populate when `hoodie.populate.meta.fields=false`. Allowed tokens are `_hoodie_commit_time` and `_hoodie_file_name`; any other token is rejected up-front. The mode is immutable at runtime (settable only at table creation, via hudi-cli, or during upgrade). Motivation: the existing all-or-nothing `populate.meta.fields=false` disables incremental queries (needs `_hoodie_commit_time`) and file-level pruning / debugging lookups (needs `_hoodie_file_name`). This lets a table opt into either or both without paying for the remaining three meta columns. Resulting modes: - populate.meta.fields=true → ALL (default; mode list ignored) - populate.meta.fields=false, mode="" → NONE - populate.meta.fields=false, mode=<subset> → selective (COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME) - populate.meta.fields=true, mode=<subset> → rejected at writer init Scope in this patch: - Spark writer path (Avro + Row): HoodieAvroParquetWriter, HoodieSparkParquetWriter, HoodieRowCreateHandle take a Set<String> meta-fields mode and populate the columns selectively; bloom filter / record-key index registration stays skipped when record key is not populated. - Incremental read path: CoW IncrementalRelationV1/V2 accept the new mode via isCommitTimePopulated(); MoR incremental relations keep the strict populate=true guard until the MoR log-write path is updated in a follow-up. Fail-fast at writer init: - Unknown token in the mode list (parser throws). - populate.meta.fields=true combined with a non-empty mode (ambiguous). - MERGE_ON_READ + non-empty mode (log-write path not yet wired; follow-up). - Non-Spark engine + non-empty mode (Flink RowData / Java-client not yet wired; follow-up). Runtime immutability: - HoodieWriterUtils.validateTableConfig now explicitly rejects a null-on-disk → non-empty transition on the mode property (existing loop only flagged the mismatch when the on-disk value was non-null, which let this silent-drop path slip through). Tests: - TestHoodieMetaFieldsMode (hudi-hadoop-common) — accessor coverage for every combination plus unknown-token rejection and whitespace tolerance. - TestHoodieWriteConfigMetaFieldsMode (hudi-client-common) — builder + validate() coverage. - TestMetaFieldsMode (hudi-spark) — end-to-end write / re-read for every mode plus the three rejection paths (populate=true+mode, unknown token, MoR+mode). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…enum Addresses review feedback on apache#19205: - Fold (populateMetaFields boolean, comma-separated mode) into a single MetaFieldsMode enum (ALL / NONE / COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME). The enum is the sole argument passed through every writer factory and writer constructor. populate.meta.fields stays on the config surface for backward compat and is auto-derived from the enum. - On-disk representation is now the enum name (e.g. "COMMIT_TIME_ONLY") instead of a comma-separated token list. Existing tables without the property continue to fall back to ALL or NONE based on the legacy hoodie.populate.meta.fields boolean. - Drop _hoodie_commit_seqno population from the selective write path — only _hoodie_commit_time is needed for incremental queries. (Review comments on HoodieRowCreateHandle:193, HoodieSparkParquetWriter:100, HoodieAvroParquetWriter:103.) - Extend the IncrementalRelation error message to state that hoodie.meta.fields.mode is physical-storage and cannot be flipped by changing write options — existing tables must be recreated to change it. - StreamSync.initializeEmptyTable() was missing the mode pass-through — wired via setMetaFieldsModeFromString. All other non-test initTable call sites either use fromProperties() (which now routes the mode through) or intentionally don't touch meta fields (CLI table create, MDT, sample writes). Test coverage additions: - TestMetaFieldsMode (hudi-spark): end-to-end assertions read parquet back and verify which meta columns are populated / null per mode. Covers row-writer path (bulk_insert with row.writer.enable=true, default) AND non-row-writer path (insert with row.writer.enable=false). Adds inline-clustering coverage for all 5 modes verifying that clustered files preserve the mode's column population semantics. - TestHoodieStreamerMetaFieldsMode (hudi-utilities): parameterized end-to-end test through the HoodieStreamer entrypoint covering all 5 modes plus MoR+selective rejection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

nsivabalan · 2026-07-13T18:12:53Z

Addressed the round of self-review comments in cfd1fea706ed (branch now rebased on latest upstream/master, force-pushed):

Fold populateMetaFields + metaFieldsMode into one argument — done. Introduced MetaFieldsMode enum (ALL / NONE / COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME) as the sole meta-field argument through every writer factory / constructor. populate.meta.fields stays on the config surface for backward compat and is auto-derived. On-disk representation is now the enum name.
Drop _hoodie_commit_seqno from selective path — done in HoodieRowCreateHandle, HoodieSparkParquetWriter, HoodieAvroParquetWriter. Only _hoodie_commit_time is populated when opted in.
Non-reversibility message on IncrementalRelation — done: extended message states the mode is physical-storage, cannot be flipped by write options, table must be recreated. The follow-up CLI PR #19206 adds table set-meta-fields-mode as the sanctioned mutation path with --force guarding tables that already have commits.

Re: "can we confirm the mode is immutable at runtime + tests across HoodieStreamer / spark datasource"

Runtime-immutability guard: HoodieWriterUtils.validateTableConfig now blocks both null → non-empty and non-empty → different transitions on hoodie.meta.fields.mode, with an error message pointing at the CLI / upgrade path (previously the loop only flagged the mismatch when the on-disk value was non-null, which let a silent-drop path slip through for older tables).
HoodieStreamer coverage: new TestHoodieStreamerMetaFieldsMode under hudi-utilities/src/test/java/.../deltastreamer/ — parameterized end-to-end test through the HoodieStreamer entrypoint covering all 5 modes plus the MoR+selective rejection assertion.
Spark datasource coverage: new TestMetaFieldsMode under hudi-spark-datasource/hudi-spark/src/test/java/.../functional/ — end-to-end assertions that read the written parquet back and verify which meta columns are populated / null per mode. Covers row-writer path (bulk_insert with row.writer.enable=true, default) AND non-row-writer path (insert with row.writer.enable=false). Also adds inline-clustering coverage for all 5 modes verifying that clustered files preserve the mode.

hudi-bot · 2026-07-13T18:19:56Z

CI report:

cfd1fea Azure: FAILURE
7b139c9 UNKNOWN

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

Previously the populateMetaFields=false branch of HoodieDatasetBulkInsertHelper projected the five Hudi meta columns as Alias(Literal(UTF8String.EMPTY_UTF8, StringType), name). A non-null Literal derives a non-null Alias, so the resulting StructField had nullable=false. Under ALL mode this was harmless because HoodieRowCreateHandle.writeRow unconditionally fills every meta slot with a non-null value. Once selective / NONE modes started leaving the opted-out columns null on the row (per hoodie.meta.fields.mode), the non-nullable StructType led SparkToParquetSchemaConverter to declare the columns as `required binary` on disk, and any subsequent read failed with ParquetDecodingException: could not read bytes at offset 0. Switch the stub to Alias(Literal.create(null, StringType), name) so the projected StructField is nullable and the physical parquet column is written as OPTIONAL. Handle-level population (writeRow / writeRowNoMetaFields / writeRowSelectiveMetaFields) is unchanged — the stub value is never observed on disk; the fix only affects the parquet schema. Fixes all row-writer + clustering selective-mode failures in TestMetaFieldsMode (commitTimeOnly / fileNameOnly / commitTimeAndFileName / none, plus clustering variants). 18/18 tests pass locally.

github-actions Bot added the size:L PR with lines of changes in (300, 1000] label Jul 6, 2026

nsivabalan commented Jul 6, 2026

View reviewed changes

github-actions Bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels Jul 6, 2026

This was referenced Jul 6, 2026

feat(cli): add table set-meta-fields-mode command #19206

Draft

feat(cli): add table set-meta-fields-mode command nsivabalan/hudi#27

Draft

nsivabalan and others added 2 commits July 13, 2026 11:00

nsivabalan force-pushed the selective_meta_fields_mode_pr_a branch from cfd1fea to 7b139c9 Compare July 13, 2026 18:12

Uh oh!

Conversation

nsivabalan commented Jul 6, 2026

Describe the issue this Pull Request addresses

Summary and Changelog

Why a list instead of a boolean or enum

Plug points

Scope

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan commented Jul 13, 2026

Uh oh!

hudi-bot commented Jul 13, 2026

CI report:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants