Repository navigation
Default to parquet in AddFiles for extension less files. - #40614
Conversation
17a68e5 to
15b03d3
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40614 +/- ##
============================================
+ Coverage 56.16% 65.07% +8.90%
- Complexity 2288 13991 +11703
============================================
Files 1124 1728 +604
Lines 178336 108508 -69828
Branches 1489 11313 +9824
============================================
- Hits 100170 70608 -29562
+ Misses 75645 33689 -41956
- Partials 2521 4211 +1690
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
5aab748 to
4a089ce
Compare
4a089ce to
f3638db
Compare
There was a problem hiding this comment.
📋 Review Summary
This PR updates Iceberg AddFiles and ReadFooterSchema to handle extensionless file paths (common in Hive outputs like 000000_0) by attempting to read their Parquet footer, while routing non-Parquet extensionless files (such as _SUCCESS markers) to the error output without failing table creation or bumping footer-read error metrics. The implementation is clean, well-structured, and thoroughly covered by both unit and integration tests.
🔍 General Feedback
- Reading the Parquet footer upfront for extensionless files before
getOrCreateTableinConvertToDataFilenicely prevents marker files like_SUCCESSfrom triggering unintended failures during initial table creation, and cachingparquetFooteravoids redundant footer reads later in the per-file processing phase. - Distinguishing
unknownFormatfromunreadableFooterinReadFooterSchemafor extensionless files ensures marker files do not emit noisy warnings or skewFOOTER_READ_ERRORS_COUNTERmetrics.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @chamikaramj for label iceberg. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Try to read files as Parquet when no extension is set (hive default behavior)