Repository navigation
WeakSet is not pickleable #74876
Description
| BPO | 30691 |
|---|---|
| Nosy | @Yhg1s, @birkenfeld, @rhettinger, @pitrou, @serhiy-storchaka |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-06-17.18:54:10.235>
labels = ['library']
title = 'WeakSet is not pickleable'
updated_at = <Date 2017-06-18.06:09:54.301>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2017-06-18.06:09:54.301>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-06-17.18:54:10.235>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 30691
keywords = []
message_count = 4.0
messages = ['296243', '296256', '296262', '296264']
nosy_count = 5.0
nosy_names = ['twouters', 'georg.brandl', 'rhettinger', 'pitrou', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue30691'
versions = []
Activity
serhiy-storchaka commentedon Jun 17, 2017
WeakSet contains the __reduce__ method, but it isn't pickleable (and never was), because the pickle state contains the value of the __dict__ dict attribute which contains a reference to unpickleable local function _remove().
I wondering whether WeakSet should be made pickleable or the __reduce__ method should be removed. __reduce__() can be used also for copying, but there are no tests for this feature.
rhettinger commentedon Jun 18, 2017
When considering whether to remove a method from long published code, if the method isn't broken, our guidance should come from whether user's have actually taken advantage of __reduce__. Determining that answer involves searching published code (Google's code search used to be a good tool for this, now we've got Github's code search tools).
In general, the burden is high for removing an existing feature (even if untested); otherwise, we risk breaking people's code for no good reason other than the joy that comes with churning code to solve an invented problem (one that has never arisen in real code and has never been reported by an actual user).
When considering whether to add pickle support, the bar is much lower. Roughly the question is amounts to balancing the potential benefits (whether someone might need to pickle a weakset someday even though we have no evidence that anyone has ever wanted this) versus the costs (risk of introducing bugs, creating cross-version incompatabilities, increasing future maintenance costs, increasing the total code volume, etc).
If adding pickling capability is easy and clean, then it seems reasonable. On the other hand, if it is even slightly tricky, we should skip adding a feature that no one has ever asked for. The
weak reference containers have long been a source of bugs, some of which were challenging to fix.
serhiy-storchaka commentedon Jun 18, 2017
The method is broken (and always was). Pickling doesn't work because the state contains unpickleable object. Copying works incorrectly, since the pickle state contains references to the original WeakSet and it overrides the __dict__ of the copy, making its state inconsistent. If the purpose of implementing the __reduce__ method was the support of the copying, the __reduce__ method should be fixed or the copying should be implemented with implementing the __copy__ and __deepcopy__ methods.
However there is a subtle moment with pickling WeakSet (as well as with pickling weakref.proxy, see bpo-18068). The problem is that if you pickled a WeakSet, but not hard references to its items, the items will be disappeared just after unpickling, since they have no hard references. This may confuse. If you pickle also hard refereces to the items, a WeakSet can be pickled and unpickled as expected (after fixing __reduce__).
rhettinger commentedon Jun 18, 2017
That seems contrary to the original intent of a WeakSet. I recommend not going down the path of adding pickle support.
If copying is also broken, then it is reasonable to either fix it or remove it depending on whether a fix is straight-forward.
serhiy-storchaka commentedon Jun 11, 2022
Copying is broken too. But instead of failing it produces incorrect result: two WeakSets share the same content.
pythongh-74876: Fix copying and pickling weakref.WeakSet
webberfc commentedon Jun 19, 2023
Hello, I am using iPyParallel (or rather, attempting to) to support parallelizing a complex task. I have successfully used iPyParallel elsewhere in another project. Unfortunately, in this project, I have this error:
I have tried to identify what the object is so I can come up with a workaround (iPyParallel allows you to instantiate remotely so pickling is not required)... iPyParallel provides two workarounds to help with pickling, being Dill and CloudPickle. Dill complains:
TypeError: no default __reduce__ due to non-trivial __cinit__And CloudPickle is complaining about a thread, which I will have to look into, I thought this version was a single thread version up until the call to map_async.
Per the above request:
I am formally requesting this :) Thanks for considering! Meanwhile I'll try to ID the object and see if I can remove it...
hugovk commentedon Apr 26, 2025
How does PR #93732 look for this?
serhiy-storchaka commentedon Jul 17, 2026
My old #93732 added support for pickling WeakSet. I wrote also extended patch to support other types of weak containers. They work fine, but with caveat -- if user pickles weak references without strong references in the same pickle, they will became dead immediately after unpickling. Weak collections became empty.
#153818 solves this problem by making the pickler to trace weak references. If they are not backed by strong references, a PicklingError is raised at the end of pickling, so such mistake will not be left unnoticed. Only
weakref.refis supported natively in thepicklemodule, all other types are expressed in terms ofweakref.ref. This does not need introducing a new opcode and bumping protocol. But this is too large change for backporting. We should just raise explicit errors in old Python versions.The current version of PR uses private functions to reconstruct. I wonder if it is worth to add public alternative constructors like
WeakRef.fromrefs().gh-74876: Support pickling weak references and weak containers (#153818)
serhiy-storchaka commentedon Oct 21, 2026
I did not merge #153818 myself, because I was not completely satisfied by it. It depends on new private API, and the pickled data is not optimal and cannot be read in older versions. I have other idea and am going to try it.
gh-74876: Skip weakref pickling tests in test_xpickle for Python < 3.…