Skip to content

[Python] Infer the right type for slices of fixed-size tuples - #40680

Merged
jrmccluskey merged 1 commit into
apache:masterfrom
raams95:fix-tuple-slice-type-inference
Oct 21, 2026
Merged

jrmccluskey merged 1 commit into
apache:masterfrom
raams95:fix-tuple-slice-type-inference

Conversation

@raams95

@raams95 raams95 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Fixes #20899

Type inference gives a slice of a fixed-size tuple the type of the whole tuple. For an element of type Tuple[str, str, int], row[1:] is inferred as Tuple[str, str, int] instead of Tuple[str, int]. This breaks pipelines, not only the reported element_type. Dropping the first field before combining fails at construction:

rows = [('day1', 'a', 1), ('day2', 'a', 2), ('day1', 'b', 3)]
with beam.Pipeline() as p:
  _ = (
      p
      | beam.Create(rows).with_output_types(Tuple[str, str, int])
      | 'DropDate' >> beam.Map(lambda row: row[1:])
      | beam.CombinePerKey(sum))
TypeCheckError: The transform 'CombinePerKey(sum)' requires PCollections of type 'Tuple[TypeVariable[K], Any]' but was applied to a PCollection of type 'Tuple[<class 'str'>, <class 'str'>, <class 'int'>]' (produced by the transform 'DropDate').

When the sliced PCollection is encoded, for example before a Reshuffle, the coder for the whole tuple is used and the pipeline fails with TypeError: '<' not supported between instances of 'str' and 'int' for row[1:], or IndexError: tuple index out of range for row[:2].

With this change, a slice of a fixed-size Tuple with constant bounds is inferred as the tuple of the selected element types (Tuple[str, int] above, Tuple[float, str, int] for [::-1] on Tuple[int, str, float]). A slice with bounds that aren't constants, such as row[i:], is inferred as Tuple[Union[...], ...] of the element types, in line with a variable index (row[i] is inferred as Union[...]) and tuple unpacking ((*a, *b)). Slices of lists, Tuple[T, ...] and str keep the type of the sliced value, as before.

Slices compile to different bytecode depending on the Python version: BUILD_SLICE followed by a subscript on 3.10 and 3.11 and for slices with a step, BINARY_SLICE on 3.12 and later, and a constant slice object on 3.14. build_slice now keeps constant bounds as a Const slice, and binary_subscr and binary_slice share one helper, so all versions infer the same types.

Tests:

  • testGetItemSlice in trivial_inference_test.py has new cases for fixed-size tuples.
  • test_inferred_type_of_tuple_slice in typed_pipeline_test.py runs the pipeline above.

Both fail before the change and pass after it on Python 3.10, 3.11, 3.12, 3.13 and 3.14.

Testing:

  • pytest apache_beam/typehints/trivial_inference_test.py apache_beam/typehints/typed_pipeline_test.py on Python 3.10, 3.11, 3.12, 3.13 and 3.14
  • pytest apache_beam/typehints apache_beam/transforms apache_beam/coders apache_beam/pipeline_test.py apache_beam/dataframe/transforms_test.py apache_beam/yaml/yaml_mapping_test.py apache_beam/yaml/yaml_udf_test.py on Python 3.12
  • yapf --diff and ruff check on the changed files

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123 or fixes #456).
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement. (Not needed, small change.)

See the Contributor Guide for more tips on how to make review process smoother.

Type inference gave a slice of a fixed-size Tuple the type of the whole
tuple, so for example row[1:] on Tuple[str, str, int] was inferred as
Tuple[str, str, int]. A following CombinePerKey then failed the type
check, and encoding the slice used the coder for the whole tuple.

Slices with constant bounds now select the element types, and other
slices of a fixed-size tuple give Tuple[Union[...], ...]. BUILD_SLICE
keeps constant bounds as a Const slice, so Python 3.10 to 3.14 infer the
same types.

Fixes apache#20899

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR fixes a bug in Python bytecode type inference where slicing a fixed-size Tuple retained the full original tuple type instead of inferring the sliced tuple type. The changes cleanly unify slice handling across BUILD_SLICE, BINARY_SLICE, and BINARY_SUBSCR (supporting Python 3.10 through 3.14+) and include comprehensive unit and pipeline tests.

🔍 General Feedback

  • The unification of slice handling via _slice_type and preserving constant bounds in build_slice as Const(slice(...)) is clean, concise, and handles both constant and dynamic slice bounds appropriately across Python versions.
  • Test coverage in trivial_inference_test.py and typed_pipeline_test.py thoroughly exercises positive/negative bounds, steps, out-of-range slices, dynamic indices, and end-to-end CombinePerKey pipeline type checking.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@codecov

codecov Bot commented Oct 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.15789% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.02%. Comparing base (76f6e6d) to head (d545485).
⚠️ Report is 242 commits behind head on master.

Files with missing lines Patch % Lines
sdks/python/apache_beam/typehints/opcodes.py 63.15% 7 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40680       +/-   ##
=============================================
+ Coverage     56.16%   80.02%   +23.85%     
=============================================
  Files          1124      490      -634     
  Lines        178336    82825    -95511     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    66278    -33892     
+ Misses        75645    16547    -59098     
+ Partials       2521        0     -2521     
Flag Coverage Δ
go ?
java ?
python 80.02% <63.15%> (+0.46%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.
Backup reviewer: damccorm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@jrmccluskey jrmccluskey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one note around the CHANGES.md file entry. Thanks!

Comment thread CHANGES.md
* (Python) DaskRunner now preserves outputs emitted by `DoFn.finish_bundle()` instead of silently dropping them ([#40519](https://github.com/apache/beam/issues/40519)).
* (Python) BigQuery streaming inserts now reset the retry budget and backoff for each batch instead of sharing them across a bundle ([#39995](https://github.com/apache/beam/pull/39995)).
* (Python) State write failures reported by portable runners now fail the bundle instead of silently losing state updates ([#39992](https://github.com/apache/beam/pull/39992)).
* (Python) Type inference now gives the right type for a slice of a fixed-size tuple, such as `row[1:]` in `beam.Map(lambda row: row[1:])`. The slice used to keep the type of the whole tuple, which could make a following `CombinePerKey` fail with a `TypeCheckError` or encode elements with the wrong coder ([#20899](https://github.com/apache/beam/issues/20899)).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a hair verbose, let's streamline a bit and let the context live in the linked bug.

Suggested change
* (Python) Type inference now gives the right type for a slice of a fixed-size tuple, such as `row[1:]` in `beam.Map(lambda row: row[1:])`. The slice used to keep the type of the whole tuple, which could make a following `CombinePerKey` fail with a `TypeCheckError` or encode elements with the wrong coder ([#20899](https://github.com/apache/beam/issues/20899)).
* (Python) Type inference now gives the right type for a slice of a fixed-size tuple, such as `row[1:]` in `beam.Map(lambda row: row[1:])`. ([#20899](https://github.com/apache/beam/issues/20899)).

@jrmccluskey
jrmccluskey merged commit e18c217 into apache:master Oct 21, 2026
87 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Type inference works incorrectly with tuple slices

2 participants