Skip to content

AddFiles: PathPattern and RouteFiles - #40749

Open
claudevdm wants to merge 2 commits into
apache:masterfrom
claudevdm:addfiles-dynamic-dest2
Open

claudevdm wants to merge 2 commits into
apache:masterfrom
claudevdm:addfiles-dynamic-dest2

Conversation

@claudevdm

Copy link
Copy Markdown
Collaborator

Adds the two self-contained pieces that dynamic destinations for IcebergIO.AddFiles are
built on: a path-pattern matcher, and the DoFn that uses it to name a destination table
from each file's path.

AddFiles registers every input file into one table named by table. Migrating a lake
usually means a tree of per-table folders, which today needs one AddFiles per table.
The goal is for one transform to route each file to the table its path names:

path_pattern:  gs://lake/export/{table}/**
table:         default.{table}

gs://lake/export/orders/part-0.parquet     ->  default.orders
gs://lake/export/customers/part-0.parquet  ->  default.customers

This PR adds only the routing primitives so they can be reviewed on their own. The
pre-pass, registration and schema-transform surface follow separately.

PathPattern

A whole-path glob with {name} placeholders, each capturing exactly one path segment.
Globbing is Beam's own (FileSystemUtils.wildcardToRegexp), so *, ** and ? behave as
they do in any other Beam file pattern; placeholders are translated to capturing groups
between the globbed fixed parts.

The whole path must match, so the pattern decides both which table a file belongs to and
whether it belongs at all. compile rejects patterns that cannot answer that
unambiguously:

rejected reason
{a} twice a name can only be captured once
{a b}, {a.b} a capture is rendered into a dotted table identifier, so no dots, slashes or blanks
gs://lake/**/{tbl}/** a ** above a placeholder leaves the captured folder undetermined
gs://lake/{tbl}/ ends with /, so it names a folder and matches no file

render(template, captures) fills the table template, and checkRenderable(template)
refuses a template naming a placeholder the pattern never captures, so table: default.{db} against .../{tbl}/** fails at construction rather than producing one error
row per file.

RouteFiles

DoFn<String, KV<String, String>> with two outputs:

  • ROUTED: KV<table, path> for a file the pattern matched
  • ERRORS: a {file, error, table} row for a path it did not, with table empty because
    there is no table to attribute it to

An unmatched path becomes an error row rather than being silently dropped: a mistyped
pattern must be visible. PathPattern.compile and checkRenderable both run in the
constructor, so a bad pattern or template fails at pipeline construction instead of once
per file on a worker.

  • RouteFiles.ERROR_SCHEMA is declared locally here. When the transform is wired up it
    becomes the same schema as AddFiles' error output, so the two can be flattened; that
    change belongs with the wiring.
  • Patterns are interpreted by Beam's Java glob translator. Note it normalises **/* to
    **, which Python's FileSystem.translate_pattern does not. Relevant only if you
    compare a path_pattern against a glob written for Python's MatchFiles upstream.

@claudevdm
claudevdm requested a review from ahmedabu98 October 23, 2026 17:14
@claudevdm
claudevdm marked this pull request as ready for review October 23, 2026 17:14
@claudevdm

Copy link
Copy Markdown
Collaborator Author

R: @ahmedabu98

@github-actions github-actions Bot added java io iceberg Iceberg IO connector size/l L: 250-999 lines changed labels Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

iceberg Iceberg IO connector io java size/l L: 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant