Skip to content

[Python] Preserve argument binding for defaulted DoFn elements - #40798

Open
xgbtxy wants to merge 2 commits into
apache:masterfrom
xgbtxy:fix/dofn-default-element-arguments
Open

xgbtxy wants to merge 2 commits into
apache:masterfrom
xgbtxy:fix/dofn-default-element-arguments

Conversation

@xgbtxy

@xgbtxy xgbtxy commented Oct 26, 2026

Copy link
Copy Markdown

Fixes #19631.

When every positional parameter has a default, _get_arg_placeholders counts the implicit element as a side-input slot. For process(element=1, timestamp=DoFn.TimestampParam, side_input=3), passing side argument 2 produces (element, 2, timestamp) instead of (element, timestamp, 2).

Skip that first ordinary default in the implicit-element branch, so the existing argument-count and placeholder logic starts at the next parameter. Explicit ElementParam and leading special-marker handling are unchanged. The same helper fixes process_batch; the tests cover that path across the existing FnApiRunner configurations. A Python bugfix note is included in CHANGES.md.

Validation on Windows/Python 3.12 with the native Cython module rebuilt using MSVC:

  • All 15 new regression/control executions pass. Before the fix, the two implicit-element positional/side-input cases and seven batch configurations fail; the six explicit/default/keyword controls pass.
  • Complete fn_runner_test.py: 465 passed, 81 skipped.
  • Complete common, core-transform, PTransform and runtime-typecheck modules with BundleBasedDirectRunner: 283 passed, 3 skipped, 5 failed. Rebuilding the exact original common.py reproduces all five failures in unchanged PTransform tests. The candidate was then restored, rebuilt and its 15 new cases rerun successfully.
  • Configured pre-commit checks pass on every changed Python file. Full-repository YAPF passes; Ruff reports the same 30 diagnostics on the original tree and candidate. Full Pyrefly, in the same activated environment, is blocked on the same two generated .pyi UTF-8 decoding errors on both versions. No unrelated lint/type-check changes were made.

The public issue was also reproduced independently on both BundleBasedDirectRunner and FnApiRunner before the change; both now return the expected values. Cloud runners and other Python versions were not run locally. AI assistance: OpenAI Codex.

Ignore the ordinary default on the implicitly supplied first input when
all positional parameters have defaults. This keeps injected parameters
and side inputs in their intended positions for process and process_batch.

Cover implicit and explicit element parameters, scalar and deferred side
inputs, and the existing FnApiRunner batch execution configurations.

Fixes apache#19631
Assisted-by: OpenAI Codex
Copilot AI balanced review requested due to automatic review settings October 26, 2026 04:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@github-actions github-actions Bot added python runners size/s S: 10-49 lines changed labels Oct 26, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR fixes issue #19631 where DoFn.process and DoFn.process_batch methods with a default value on the implicit first (element/batch) parameter misbound subsequent special DoFn parameters and side inputs. By stripping the leading ordinary default in _get_arg_placeholders when all positional parameters have defaults, the element slot and remaining parameter placeholders align properly.

🔍 General Feedback

  • The fix in sdks/python/apache_beam/runners/common.py is minimal, well-scoped, and preserves existing handling for explicit DoFn.ElementParam and special parameter markers (_DoFnParam and BundleFinalizerParam).
  • Test coverage in common_test.py and fn_runner_test.py thoroughly exercises default, positional, keyword, and side-input bindings for both process and process_batch.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @damccorm for label python.
Backup reviewer: tvalentyn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

not isinstance(default_arg_values[0], core._DoFnParam) and
default_arg_values[0] is not core.DoFn.BundleFinalizerParam):
# The implicit element occupies the first slot even if it has a default.
default_arg_values = default_arg_values[1:]

@damccorm damccorm Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: For a single-parameter def process(self, element=1):, slicing default_arg_values[1:] produces [], so arg_names[-len(default_arg_values):] on line 736 evaluates to arg_names[-0:] (['element']). zip(..., []) still yields nothing today, though guarding the loop on line 736 with if default_arg_values: would avoid the -0 slice pitfall if strict=True is ever added later.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the if default_arg_values: guard in 0dbcf73. This skips the -0 slice when no defaults remain.

@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.90%. Comparing base (2213e54) to head (52cc6f2).
⚠️ Report is 27 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #40798       +/-   ##
===========================================
+ Coverage        0   79.90%   +79.90%     
===========================================
  Files           0      493      +493     
  Lines           0    83475    +83475     
===========================================
+ Hits            0    66704    +66704     
- Misses          0    16771    +16771     
Flag Coverage Δ
python 79.90% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@damccorm

Copy link
Copy Markdown
Contributor

waiting on author

Skip the default-argument loop when no defaults remain, avoiding
the negative-zero slice noted in review.

Assisted-by: OpenAI Codex
@github-actions github-actions Bot added size/m M: 50-249 lines changed Next Action: Reviewers and removed size/s S: 10-49 lines changed Next Action: Author labels Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DoFn invocation has wrong argument assignment

3 participants