Repository navigation
[Python] Handle NaN in DataFrame drop_duplicates, duplicated and nunique - #40571
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40571 +/- ##
==========================================
- Coverage 80.04% 79.93% -0.12%
==========================================
Files 490 493 +3
Lines 82916 83414 +498
==========================================
+ Hits 66373 66678 +305
- Misses 16543 16736 +193
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| * (Python) Fixed stateful DoFns with side inputs sometimes taking the timer key coder from a side input instead of the main input, which could make the worker fail to decode timer keys with `Unknown type tag` ([#40374](https://github.com/apache/beam/issues/40374)). | ||
| * (Python) `Duration` built from float seconds now rounds to the nearest microsecond instead of truncating, which could lose a microsecond ([#40263](https://github.com/apache/beam/issues/40263)). | ||
| * (Python) `ReadFromText` with `escapechar` no longer skips a delimiter that directly follows an escaped delimiter, which merged two records into one ([#40459](https://github.com/apache/beam/issues/40459)). | ||
| <<<<<<< ours |
There was a problem hiding this comment.
Conflict in CHANGES.md needs to be resolved.
There was a problem hiding this comment.
Sorry, conflict markers slipped in when I rebased. Rebased on master and fixed in 4d80dbe.
There was a problem hiding this comment.
The tests on master was broken last weekend. Could you merge to the master again?
There was a problem hiding this comment.
Rebased on current master (4689e16), kept both CHANGES.md entries.
There was a problem hiding this comment.
Thanks! I am re-triggering the failed tests.
37f569f to
4d80dbe
Compare
4d80dbe to
4689e16
Compare
shunping
left a comment
shunping
left a comment
There was a problem hiding this comment.
LGTM! Will merge after the tests are passed.
Fixes #40570
drop_duplicatesandduplicatedcolocated duplicates withgroupby, which drops NaN keys (anddropna=Falsedoesn't work with multiple columns, #21014). They now add a hash of the compared columns as an index level and partition on that, so duplicates still end up together. -0.0 is normalized first since pandas treats it as a duplicate of 0.0 but it hashes differently.Series.nuniquenow honoursdropna.Added tests to frames_test.py (they fail on master with
KeyError) and a CHANGES.md entry. frames_test.py and pandas_doctests_test.py pass.Written with help from Claude.