Add ParquetExporter (Vortex -> Parquet) - #362
Merged
Merged
Conversation
dfa1
force-pushed
the
feat/parquet-exporter
branch
3 times, most recently
from
September 3, 2026 04:39
4241c98 to
f3bd665
Compare
Inverse of ParquetImporter, built on Hardwood 1.1's new ColumnWriter/
ParquetFileWriter write API. Flat schemas only: Bool, non-F16
Primitive, Utf8, Binary, and vortex.timestamp (MILLIS/MICROS/NANOS);
Struct/List/Map top-level columns throw UnsupportedOperationException.
- Bumps dev.hardwood:hardwood-core 1.0.0.Final -> 1.1.0.Beta1 (adds the
write API this depends on, plus read-path speedups for ParquetImporter).
- ParquetExporter / ExportOptions in the parquet module, mirroring
ParquetImporter/ImportOptions' shape.
- vortex-reader promoted from test- to production-scope dependency in
parquet/pom.xml.
- ParquetExporterTest: type-mapping unit tests, flat export + projection
+ unsupported-type integration tests, and two round-trip tests
(Vortex -> Parquet -> Vortex via ParquetImporter) covering flat types
(including U8/U32 boundary values) and the timestamp extension.
- CLI: the `export` subcommand now dispatches to ParquetExporter when the
output path ends `.parquet` (previously CSV-only); usage text and
ExportCommandTest updated.
- Remote HTTP(S) support, both directions:
- ParquetExporter.exportParquet(VortexHandle, Path[, ExportOptions]):
exports an already-open handle (local VortexReader or remote
VortexHttpReader) without an intervening local copy; the handle is
not closed here, the caller keeps ownership.
- ParquetImporter.importParquet(URI, Path[, ImportOptions]): imports a
Parquet file served over HTTP(S), fetched entirely through targeted
Range requests via the new HttpInputFile (implements Hardwood's
InputFile), mirroring VortexHttpReader's own range-fetch pattern. No
full-file download occurs.
- checkNoDuplicateNames/fillRow take a String sourceName instead of a
Path, since a remote source has no filesystem path to report in
error messages.
- CLI: `export`/`import` accept a url source too (Parquet-only in both
directions; CSV and stdout streaming from a URL are out of scope).
- HttpInputFileTest covers the HEAD/Range request mechanics against a
mocked HttpClient. Live-verified end to end against a real ~3M-row
public Parquet file (NYC taxi trip data) via the built CLI jar:
remote Parquet -> local Vortex -> local Parquet -> reimport, with
row counts and min/max stats matching exactly at every step.
- docs/reference.md, docs/how-to.md: document the new class/options, the
remote overloads, and the CLI's Parquet import/export support.
- DocsConsistencyTest: allowlist WriterConfig as a known external
receiver cited in reference.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LjU7zWghUiJauRxK58nyUs
dfa1
force-pushed
the
feat/parquet-exporter
branch
from
September 3, 2026 04:55
f3bd665 to
3d38a8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ParquetExporter/ExportOptionsin theparquetmodule — the inverse ofParquetImporter— writing a Vortex file to Parquet via Hardwood 1.1's newColumnWriter/ParquetFileWriterwrite API.Bool, non-F16Primitive,Utf8,Binary, andvortex.timestamp(MILLIS/MICROS/NANOS).Struct/List/Maptop-level columns throwUnsupportedOperationException— nested export is a documented follow-up.dev.hardwood:hardwood-core1.0.0.Final → 1.1.0.Beta1 (adds the write API this depends on, plus read-path speedups forParquetImporter).docs/reference.md,docs/how-to.md,CHANGELOG.md.Test plan
./mvnw verify -pl parquet -am— unit tests (29 inParquetExporterTest: type mapping, flat export + projection + unsupported-type rejection, and two Vortex→Parquet→Vortex round-trip tests viaParquetImporter, covering U8/U32 boundary values and the timestamp extension)./mvnw verify— full reactor build, all modules green./mvnw verify -pl integration -am -Dit.test=DocsConsistencyTest— living-docs claims still resolve🤖 Generated with Claude Code
https://claude.ai/code/session_01LjU7zWghUiJauRxK58nyUs