Closed
Labels
area:Icebergarea:writerNative Parquet writerNative Parquet writerdocumentationImprovements or additions to documentationImprovements or additions to documentationrequires-triage
Description
What is the problem the feature request solves?
The Iceberg user guide has errors and gaps about native writes:
docs/source/user-guide/latest/iceberg-writes.md:273-276says float/double partition directories use Rust's shortest representation (f=1where iceberg-java writesf=1.0). Since fix: render float and double Iceberg partition values like iceberg-java #5840 they use Java'sDouble.toStringrules and match iceberg-java.iceberg-writes.md:316-319says the parity suite compares committed manifests byte-for-byte.CometIcebergWriteActionSuitecompares aggregatedreadable_metrics(min/max, counts), not manifest bytes.iceberg-writes.mddoes not say that writes whose input is a local relation, such asINSERT ... VALUESanddf.writeTo(...).append()on a local DataFrame, needspark.comet.exec.localTableScan.enabled(defaultfalse). Without it they silently use iceberg-java even with both write flags on.iceberg-writes.mddoes not list Native Iceberg writer keeps a dictionary page for high-cardinality columns where iceberg-java writes none #6114 (dictionary page kept for high-cardinality columns) among the accepted divergences.docs/source/user-guide/latest/iceberg.md:184lists "Iceberg writes (reads are accelerated, writes use Spark)" as unsupported. Native writes exist behindspark.comet.iceberg.write.enabled; that page should point toiceberg-writes.md.
Describe the potential solution
Fix the four points above. The "experimental, disabled by default" wording will need another pass when the defaults change (#5644).
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649.
Metadata
Metadata
Assignees
Labels
area:Icebergarea:writerNative Parquet writerNative Parquet writerdocumentationImprovements or additions to documentationImprovements or additions to documentationrequires-triage
Activity
docs: fix stale and missing native Iceberg write details (apache#6150)