Skip to content

Preserve all streams when interleave inputs share an iterator - #641

Open
fhgffy wants to merge 1 commit into
pytoolz:masterfrom
fhgffy:fix/interleave-shared-iterators
Open

fhgffy wants to merge 1 commit into
pytoolz:masterfrom
fhgffy:fix/interleave-shared-iterators

Conversation

@fhgffy

@fhgffy fhgffy commented Oct 2, 2026

Copy link
Copy Markdown

interleave can silently drop values from unrelated inputs when the same iterator occupies more than one input slot:

shared = iter([0, 1])
list(interleave([shared, shared, [10, 11]]))
# Current: [0, 1, 10]
# Expected: [0, 1, 10, 11]

If the shared iterator starts empty, the entire [10, 11] input is lost. The removal loop uses iterator identity to find the exhausted slot again, so another occurrence of that iterator ends the retained cycle too early.

Wrap each input occurrence in a distinct single-input itertools.chain. This preserves shared consumption while giving each slot its own identity. The existing cycle/removal algorithm and inner loop remain intact, including support for infinite inner and outer inputs.

The three regression tests cover adjacent and separated shared slots, empty and odd-length shared inputs, and infinite inputs. All three fail before the change and pass afterward.

Validation on Linux:

  • CPython 3.12.14 and 3.13.5: full package tests and doctests each pass with 271 passed, 2 skipped
  • pytest bench/ -q: 19 passed
  • Repository CI's pycodestyle command and git diff --check: passed
  • Separate round-robin oracle checks passed for 15,125 finite shared-iterator configurations, plus consumption/laziness and exception-propagation checks
  • Other Python versions and platforms were not run locally

Performance tradeoff: the wrapper adds per-input overhead, and timings vary by workload. One local median-of-five sample for ten streams of 1,000 elements (number=200) took 0.0751 s before and 0.0930 s after, about 24% slower. Uneven-length and many-short-stream samples were close to baseline (ratios 0.963 and 0.995). This fixes data loss but does not claim performance neutrality.

@fhgffy
fhgffy marked this pull request as ready for review October 2, 2026 13:00

@Wkhanex Wkhanex left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked 38234a3 against a separate deque-of-slots round-robin oracle for 7,680 finite configurations: three source lengths in 0–3 and every 1–4-slot arrangement referring to those sources. That includes adjacent/separated aliases and empty shared sources. All outputs matched; the base loses an unrelated source in an empty-alias case.

An additional check put a RuntimeError in an aliased source after its first item: the two earlier values were yielded in order and the original exception object propagated from the next shared slot. All 7,683 assertions and the four existing interleave tests pass on CPython 3.12.14. I did not benchmark the wrapper overhead or run the complete suite.

Review prepared with Codex assistance; the checks described above were executed locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants