Parquet: Fix nested initial default applied when ancestor struct is null by amogh-jahagirdar · Pull Request #17320 · apache/iceberg

amogh-jahagirdar · 2026-07-21T14:53:01Z

When a query projects only a nested field that has an initial default and doesn't include the struct's real fields, PruneColumns reduces the struct to an empty group (which I think is correct, expected behavior for that abstraction). That discards the file leaf columns whose definition levels carry if the struct itself is null or not, so a null struct was incorrectly materialized as a struct of default values. in SQL

select struct.field from t1...

where struct is null but struct.field has a default value would surface the default value instead of null, which is not spec compliant.

To address this we solve this in two parts:

Retain 1 leaf that we probe for (even if it's not being projected) just so we have a definition level to work with. I don't think there's any other way to get definition levels because those are carried at the very leaf level of a struct.
The default value constant reader uses that real leafs definition level to derive if the parent struct is null or not so that we surface the spec compliant results.

amogh-jahagirdar · 2026-07-21T14:55:09Z

Spark/Flink tests will fail because their row oriented readers still need to be updated to have a similar fix as to what's done in BaseParquetReader. But I wnated to talk through the solution first with others.

pvary · 2026-07-23T15:22:03Z

      for (Types.NestedField field : expectedFields) {
        int id = field.fieldId();
        ParquetValueReader<?> reader =
            ParquetValueReaders.replaceWithMetadataReader(
                id, readersById.get(id), idToConstant, constantDefinitionLevel);
-        reorderedFields.add(defaultReader(field, reader, constantDefinitionLevel));
+        boolean hostsProbe = probeHostId != null && id == probeHostId;
+        reorderedFields.add(


This will this mess up structs which only contain constant fields?

Schema readSchema = new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional("nested") .withId(2) .ofType( Types.StructType.of( Types.NestedField.optional(4, "part", Types.StringType.get()))) .build());

The AI wrote this unit test for me:

@Test public void testNestedConstantWhenAncestorStructIsNull() throws IOException { // one row has an inner value, the other has a null nested struct Schema writeSchema = new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional("nested") .withId(2) .ofType( Types.StructType.of( Types.NestedField.required(3, "inner", Types.StringType.get()))) .build()); Record present = GenericRecord.create(writeSchema); present.setField("id", 1L); Record presentNested = GenericRecord.create(writeSchema.findField("nested").type().asStructType()); presentNested.setField("inner", "a"); present.setField("nested", presentNested); Record nullNested = GenericRecord.create(writeSchema); nullNested.setField("id", 2L); nullNested.setField("nested", null); OutputFile output = new InMemoryOutputFile(); try (FileAppender<Record> appender = Parquet.write(output) .schema(writeSchema) .createWriterFunc(GenericParquetWriter::create) .build()) { appender.add(present); appender.add(nullNested); } // project only a metadata/partition constant field (served from idToConstant); inner is dropped Schema readSchema = new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional("nested") .withId(2) .ofType( Types.StructType.of( Types.NestedField.optional(4, "part", Types.StringType.get()))) .build()); Map<Integer, ?> idToConstant = ImmutableMap.of(4, "US"); List<Record> rows; try (CloseableIterable<Record> reader = Parquet.read(output.toInputFile()) .project(readSchema) .createReaderFunc( fileSchema -> GenericParquetReaders.buildReader(readSchema, fileSchema, idToConstant)) .build()) { rows = Lists.newArrayList(reader); } assertThat(rows).hasSize(2); // present struct reads the constant Record row1Nested = (Record) rows.get(0).getField("nested"); assertThat(row1Nested).isNotNull(); assertThat(row1Nested.getField("part")).isEqualTo("US"); // null struct reads as null assertThat(rows.get(1).getField("nested")).isNull(); }

Constants beyond default values also already had this issue, it's not a new regression in this change. But I'll see if it's easy to handle that case as well

I thought about this scenario some more and I don't think it's valid because the only condition where the idToConstant map has something for a nested source is identity partitions.

But in that case it would be impossible to have a non-null idToConstant and a null value in the file, because that violates the partitioning requirement (a null nested field means the row belongs to the null partition, not the "US" partition). The opposite situation (null idToConstant and non-null partition value materialized in the file) would also not be allowed. The metadata columns like row ID etc. are all top level IDs, so they never show up nested inside a struct so the fix isn't relevant for that.

Non-identity transforms like day/bucket/truncate don't come into play here either, since constantsMap only puts a source id into idToConstant when the transform is identity.

So the initial default case is really the only one that needs this behavior, which is what the PR handles.

Let me know what you think @pvary

Yeah, that's a good point.
Also checked that if a Spark query results in a struct, and it contains the metadata columns, then it is converted to straight column reads.

So, yeah, you are correct. We are fine with this fix.

pvary · 2026-07-23T15:22:53Z

Maybe we should test based on the spec example:

Default values for fields that are nested structs must not contain default values for the struct's fields (sub-fields).

For example, a struct column point with fields x (default 0) and y (default 0) can be defaulted to {"x": 0, "y": 0} or null. A non-null default is stored by setting initial-default or write-default to an empty struct ({}) that will use field values set from each field's initial-default or write-default, respectively.

If I understand correctly, the {} default is not working - this might worth a different PR as the {} is currently rejected.

Tishj · 2026-07-23T18:12:19Z

Maybe we should test based on the spec example:

Default values for fields that are nested structs must not contain default values for the struct's fields (sub-fields).

For example, a struct column point with fields x (default 0) and y (default 0) can be defaulted to {"x": 0, "y": 0} or null. A non-null default is stored by setting initial-default or write-default to an empty struct ({}) that will use field values set from each field's initial-default or write-default, respectively.

If I understand correctly, the {} default is not working - this might worth a different PR as the {} is currently rejected.

#16596

pvary · 2026-08-19T12:57:31Z

+     * Returns the descriptor for the first leaf column in the file that is nested under the given
+     * struct path, or null if the struct has no leaf columns in the file.
+     */
+    private ColumnDescriptor firstLeafUnder(String[] structPath) {


Seems like duplicated methods in Flink/Spark. Could we find a better place?

pvary · 2026-08-24T13:49:17Z

+      }
+
+      GroupType fileFieldGroup = fileField.asGroupType();
+      if (isListOrMap(fileFieldGroup)) {


I think we need to use an implementation of TypeWithSchemaVisitor, since we can have a array<struct<...>> where the struct has a field with default value.

pvary · 2026-08-24T14:57:51Z

+   * Returns the first leaf column under the struct, or null if an expected field already reads a
+   * file column or the struct has no leaf columns.
+   */
+  private static ColumnDescriptor definitionLevelProbe(


We should make this align with firstNonNullColumn too

Done, but what I did was make firstNonNullColumn choose a real materialized column instead of the constant reader definition level. Though to be clear, this is purely just to be defensive and robust to any change in how constant readers may work in the future. the constant reader scenario isn't quite possible today because it's only used for identity partitions, and that's not possible to both be null AND have some other value as it goes against the partitioning definition.

But I agree though it is best to write this in a manner which doesn't make any assumptions on how it's being used today.

amogh-jahagirdar · 2026-09-03T14:54:09Z

+   * Returns the first leaf column under the struct, or null if an expected field already reads a
+   * file column or the struct has no leaf columns.
+   */
+  private static ColumnDescriptor definitionLevelProbe(


Done, but what I did was make firstNonNullColumn choose a real materialized column instead of the constant reader definition level. Though to be clear, this is purely just to be defensive and robust to any change in how constant readers may work in the future. the constant reader scenario isn't quite possible today because it's only used for identity partitions, and that's not possible to both be null AND have some other value as it goes against the partitioning definition.

But I agree though it is best to write this in a manner which doesn't make any assumptions on how it's being used today.

amogh-jahagirdar · 2026-09-03T14:57:28Z

@@ -182,4 +184,154 @@ public void testTwoLevelList() throws IOException {
      assertThat(Lists.newArrayList(reader)).hasSize(1);
    }
  }
+
+  @Test


i'm a little skeptical this test is needed now, feels a bit duplicative

I did keep a more "representative" test but dropped the other one; the rationale for keeping the more realistic test for generic reader was that even though all of the other tests share the same Parquet reader logic, there's still some logic in the different implementations which adapt to the engine record type (e.g. FlinkParquetReaders/ SparkParquetReaders) which compute things like readersById and invoke the newly added structFieldReader.

The Generic case is another implementation of that and we'd want to catch any bugs in that middle layer that may only apply for generic reader but don't impact the Spark/Flink ones.

When a query projects only a nested field that has an initial default (or a metadata/partition constant) and drops the struct's real fields, PruneColumns reduces the struct to an empty group. That discards the file leaf columns whose definition levels carry the struct's per-row null-ness, so a null struct was incorrectly materialized as a struct of default values. To address this we solve this in two parts: 1. Retain 1 leaf that we probe for (even if it's not being projected) just so we have a definition level to work with 2. The default value constant reader uses that real leafs definition level to derive if the parent struct is null or not so that we surface the spec compliant results.

pvary · 2026-09-04T09:43:43Z

@@ -129,12 +131,94 @@ public static Type fieldType(GroupType group, String name) {

  public static MessageType pruneColumns(MessageType fileSchema, Schema expectedSchema) {


This method is called from ParquetReadSupport.init which is used by parquet-mr. There the returned reader context returns the newly projected extra columns and these columns are returned if no createReaderFunc has been set.

If this is a valid, supported scenario (parquet-mr, no createReaderFunc), then we should fix it as the returned values will change.

This could help to repro in TestGenericData:

private static final Schema NESTED_MULTI_LEAF_SCHEMA = new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional( 2, "nested", Types.StructType.of( Types.NestedField.required(3, "big", Types.StringType.get()), Types.NestedField.required(4, "inner", Types.StringType.get())))); private static final Schema NESTED_DEFAULT_ONLY_READ_SCHEMA = new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional("nested") .withId(2) .ofType( Types.StructType.of( Types.NestedField.optional("added") .withId(9) .ofType(Types.StringType.get()) .withInitialDefault(Literal.of("US")) .build())) .build()); @Test public void testGenericReadDoesNotMaterializeUnprojectedColumns() throws IOException { Types.StructType nestedType = NESTED_MULTI_LEAF_SCHEMA.findField("nested").type().asStructType(); Record present = GenericRecord.create(NESTED_MULTI_LEAF_SCHEMA); present.setField("id", 1L); Record presentNested = GenericRecord.create(nestedType); presentNested.setField("big", "b"); presentNested.setField("inner", "a"); present.setField("nested", presentNested); Record nullNested = GenericRecord.create(NESTED_MULTI_LEAF_SCHEMA); nullNested.setField("id", 2L); nullNested.setField("nested", null); OutputFile output = new InMemoryOutputFile(); try (FileAppender<Record> appender = Parquet.write(output) .schema(NESTED_MULTI_LEAF_SCHEMA) .createWriterFunc(GenericParquetWriter::create) .build()) { appender.add(present); appender.add(nullNested); } // the generic read path has no reader function and materializes the pruned projection directly List<org.apache.avro.generic.GenericRecord> rows; try (CloseableIterable<org.apache.avro.generic.GenericRecord> reader = Parquet.read(output.toInputFile()).project(NESTED_DEFAULT_ONLY_READ_SCHEMA).build()) { rows = Lists.newArrayList(reader); } assertThat(rows).hasSize(2); for (org.apache.avro.generic.GenericRecord row : rows) { org.apache.avro.generic.GenericRecord nested = (org.apache.avro.generic.GenericRecord) row.get("nested"); if (nested != null) { assertThat(nested.getSchema().getFields()) .extracting(org.apache.avro.Schema.Field::name) .as("read records must not expose columns outside the projection") .doesNotContain("big", "inner"); } } }

I'm not sure this is a valid case, or at least a case we need to concern ourselves with in my opinion. The parquet read path where a reader function is not passed in, isn't invoked anywhere within the codebase and all the engines pass through a reader function. That path never supported default values let alone this specific handling of a default value case. Though it maybe a good idea in a separate PR to block reads through that path if there are initial default values and a reader function isn't specified, but considering it's been there and hasn't really been a practical issue I'd prefer to leave it for a separate PR>

pvary · 2026-09-04T10:29:39Z

+    List<ColumnDescriptor> columns = Lists.newArrayList();
+    for (ColumnDescriptor column : fileSchema.getColumns()) {
+      // a probe leaf is kept in the read set by id, so a leaf without one cannot be a probe
+      if (column.getPrimitiveType().getId() == null) {


We need to handle the repetition level here. Otherwise leafs inside maps or lists not handled correctly.

Could be checked by this in ReadFormatModelTests:

@ParameterizedTest @FieldSource("FILE_FORMATS") void testNestedDefaultValueWhenParentStructWithListIsNull(FileFormat fileFormat) throws IOException { assumeSupports(fileFormat, FEATURE_READER_DEFAULT); Types.NestedField idField = Types.NestedField.required(1, "id", Types.LongType.get()); Types.NestedField nestedField = Types.NestedField.optional("nested") .withId(2) .ofType( Types.StructType.of( Types.NestedField.optional( 3, "tags", Types.ListType.ofRequired(4, Types.StringType.get())), Types.NestedField.required(5, "inner", Types.StringType.get()))) .build(); Schema writeSchema = new Schema(idField, nestedField); Types.StructType nestedType = nestedField.type().asStructType(); // alternate present and null structs, with a different number of list elements per row so that // a reader tracking the list column would fall behind by more than one value per row List<Record> genericRecords = Lists.newArrayList(); for (int i = 0; i < 5; i += 1) { Record record = GenericRecord.create(writeSchema); record.setField("id", (long) i); if (i % 2 == 0) { Record nested = GenericRecord.create(nestedType); nested.setField("tags", IntStream.range(0, i).mapToObj(j -> "tag-" + j).toList()); nested.setField("inner", "inner-" + i); record.setField("nested", nested); } genericRecords.add(record); } writeGenericRecords(fileFormat, writeSchema, genericRecords); Schema expectedSchema = schemaWithOnlyDefaultedNestedField(); readAndAssertEngineRecords( fileFormat, expectedSchema, genericRecords, defaultedNestedRecord(expectedSchema)); } /** * Projects "nested" with a single added field that is missing from the file but has a default. */ private static Schema schemaWithOnlyDefaultedNestedField() { return new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional("nested") .withId(2) .ofType( Types.StructType.of( Types.NestedField.optional("added") .withId(100) .ofType(Types.StringType.get()) .withInitialDefault(Literal.of("US")) .build())) .build()); } private static Function<Record, Record> defaultedNestedRecord(Schema expectedSchema) { Types.StructType nestedType = expectedSchema.findField("nested").type().asStructType(); return record -> { Record expected = GenericRecord.create(expectedSchema); expected.setField("id", record.getField("id")); if (record.getField("nested") != null) { Record expectedNested = GenericRecord.create(nestedType); expectedNested.setField("added", "US"); expected.setField("nested", expectedNested); } return expected; }; }

Great catch, to address this I made the leaf probe selection only pick a leaf whose repetition level matches the struct's own, so no LIST/MAP sits between them, and shared that logic between schema pruning and reader construction so they are consistent.

Since probe candidates have specific requirements that may not match the schema, we bias towards failing the read rather than silently falling back to the old broken behavior, so I added Preconditions checks for that (e.g. a struct whose only fields are lists/maps, no flat leaf anywhere)."

This does mean that there maybe some schemas + files with initial default values that may fail to read but this is better than returning incorrect results. For example, a nullable struct whose only fields are tags: list and tags2: map<string,string> (no flat scalar anywhere), read with a schema that only projects a new added field with an initial default: there's no leaf left to check whether the struct itself is null, so we now fail that read instead of incorrectly applying the default to rows where the struct is actually null.

I'm not sure if there's any better way of handling this case but at least now it should be correct though very specific cases we'd fail on because we couldn't select a candidate.

pvary · 2026-09-04T11:00:07Z

+      Object value = convertConstant.apply(field.type(), field.initialDefault());
+      return probe != null ? constant(value, probe) : constant(value, constantDefinitionLevel);
+    } else if (field.isOptional()) {
+      return nulls();


If we have a nested struct which contains not projected fields, and the projcted field has no default, then we should return Racord(null), and not null.

Could be tested with a test in ReadFormatModelTest like:

@ParameterizedTest @FieldSource("FILE_FORMATS") void testNestedProjectionWithoutDefaultWhenParentStructIsNull(FileFormat fileFormat) throws IOException { assumeSupports(fileFormat, FEATURE_READER_DEFAULT); Types.NestedField idField = Types.NestedField.required(1, "id", Types.LongType.get()); Types.NestedField nestedField = Types.NestedField.optional( 2, "nested", Types.StructType.of(Types.NestedField.required(3, "inner", Types.StringType.get()))); Schema writeSchema = new Schema(idField, nestedField); Record present = GenericRecord.create(writeSchema); present.setField("id", 1L); Record presentNested = GenericRecord.create(nestedField.type().asStructType()); presentNested.setField("inner", "a"); present.setField("nested", presentNested); Record nullNested = GenericRecord.create(writeSchema); nullNested.setField("id", 2L); nullNested.setField("nested", null); List<Record> genericRecords = List.of(present, nullNested); writeGenericRecords(fileFormat, writeSchema, genericRecords); // the only projected field of "nested" is missing from the file and has no initial default, so // a present struct must still be read as a struct with a null field, not as a null struct Schema expectedSchema = new Schema( idField, Types.NestedField.optional( 2, "nested", Types.StructType.of( Types.NestedField.optional(4, "added", Types.StringType.get())))); Types.StructType expectedNestedType = expectedSchema.findField("nested").type().asStructType(); readAndAssertEngineRecords( fileFormat, expectedSchema, genericRecords, record -> { Record expected = GenericRecord.create(expectedSchema); expected.setField("id", record.getField("id")); if (record.getField("nested") != null) { expected.setField("nested", GenericRecord.create(expectedNestedType)); } return expected; }); }

Hm, this maybe a separate issue. The test doesn't set up an initial default though the failure in the test does indicate some other issue that's unrelated to default values.

This issue looks like a pre-existing gap in how a struct's own presence gets tracked when none of its projected fields end up reading anything real from the file, whether that's because a field has no default or just because it's a plain missing optional field doesn't matter, the struct reader has nothing left to check against the file either way.

e.g.

Example. File has two rows: row 1: id=1, nested={inner: "a"} <- nested is present row 2: id=2, nested=null <- nested is null You read with a projection where nested only asks for a field added that doesn't exist in the file, and has no default. Expected: row 1: nested = {added: null} (nested was present, just has nothing for "added") row 2: nested = null Actual (the bug): row 1: nested = null <- WRONG, should be {added: null} row 2: nested = null (this one happens to be right)

I think it's worth its own issue rather than folding into this one since it's independent of default values and seems like some long standing pre-existing behavior. Wdyt @pvary ?

pvary caught this: the leaf we borrow to signal a constant-only struct's null-ness could sit behind a LIST/MAP, so it doesn't produce one value per struct occurrence and desyncs the reader. Share the selection logic between schema pruning and reader construction so they agree on the same leaf, restrict it to repetition-safe leaves, and fail loudly instead of silently falling back when none exists.

github-actions Bot added parquet data labels Jul 21, 2026

amogh-jahagirdar requested a review from rdblue July 21, 2026 14:54

amogh-jahagirdar force-pushed the fix-nested-default-null-struct branch from 9a0a571 to 5adf95b Compare July 21, 2026 14:56

amogh-jahagirdar requested review from nastra, pvary and stevenzwu July 21, 2026 16:29

pvary reviewed Jul 23, 2026

View reviewed changes

nssalian added this to the Iceberg 1.12.0 milestone Aug 18, 2026

github-actions Bot added spark flink labels Aug 19, 2026

amogh-jahagirdar force-pushed the fix-nested-default-null-struct branch from 1048894 to 936efed Compare August 19, 2026 03:51

pvary reviewed Aug 19, 2026

View reviewed changes

amogh-jahagirdar force-pushed the fix-nested-default-null-struct branch from 8e8ef58 to 4c6c948 Compare August 19, 2026 14:31

pvary reviewed Aug 24, 2026

View reviewed changes

amogh-jahagirdar commented Sep 3, 2026

View reviewed changes

amogh-jahagirdar force-pushed the fix-nested-default-null-struct branch from 77d74b3 to cdba2fb Compare September 3, 2026 15:37

nssalian requested a review from pvary September 3, 2026 22:04

pvary reviewed Sep 4, 2026

View reviewed changes

amogh-jahagirdar force-pushed the fix-nested-default-null-struct branch from 8858ea7 to c4a182d Compare September 6, 2026 15:18

amogh-jahagirdar requested a review from pvary September 6, 2026 15:24

		@@ -129,12 +131,94 @@ public static Type fieldType(GroupType group, String name) {

		public static MessageType pruneColumns(MessageType fileSchema, Schema expectedSchema) {

Uh oh!

Conversation

amogh-jahagirdar commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amogh-jahagirdar commented Jul 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvary commented Jul 23, 2026

Uh oh!

Tishj commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvary Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amogh-jahagirdar commented Jul 21, 2026 •

edited

Loading

amogh-jahagirdar Jul 26, 2026 •

edited

Loading

amogh-jahagirdar Jul 26, 2026 •

edited

Loading

amogh-jahagirdar Sep 6, 2026 •

edited

Loading

pvary Sep 4, 2026 •

edited

Loading

amogh-jahagirdar Sep 6, 2026 •

edited

Loading