function changed when pickle bound method object

[georgexsh]

BPO	37297
Nosy	@rhettinger, @pitrou, @tiran, @asvetlov, @serhiy-storchaka, @maggyero
Files	0001-fix-bound-method-reduce-bug.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-06-16.01:40:57.101>
labels = ['3.7', 'type-bug', 'library']
title = 'function changed when pickle bound method object'
updated_at = <Date 2019-06-17.21:01:20.919>
user = 'https://bugs.python.org/georgexsh'

bugs.python.org fields:

activity = <Date 2019-06-17.21:01:20.919>
actor = 'brett.cannon'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-06-16.01:40:57.101>
creator = 'georgexsh'
dependencies = []
files = ['48424']
hgrepos = []
issue_num = 37297
keywords = ['patch']
message_count = 4.0
messages = ['345721', '345776', '345867', '345872']
nosy_count = 7.0
nosy_names = ['rhettinger', 'pitrou', 'christian.heimes', 'asvetlov', 'serhiy.storchaka', 'georgexsh', 'maggyero']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37297'
versions = ['Python 3.7']

[georgexsh]

if we create a bound method object f with function object A.f and instance object B(),
when pickling this bound method object:

	import pickle

	class A():
	    def f(self):
	        pass

	class B():
	    def f(self):
	        pass

	o = B()
	f = A.f.__get__(o)
	pf = pickle.loads(pickle.dumps(f))
	print(f)
	print(pf)

we get:

<bound method A.f of <__main__.B object at 0x10b82ac18>>
<bound method B.f of <__main__.B object at 0x10b82ac88>>

the underlaying function are lost, changed from A.f to B.f.

as pickle calls __reduce__ method of method object, IMO its implementation simply ignored the real function, whcih is not right.

I have tried a wordaround:

	import types
	import copyreg

	def my_reduce(obj):
	    return (obj.__func__.__get__, (obj.__self__,))

	copyreg.pickle(types.MethodType, my_reduce)

[maggyero]

As you stated on Stackoverflow, the fact that the function A.f becomes the function B.f after a pickling-unpickling sequence creates an infinite recursive call of B.f in worker processes started with multiprocessing.pool.Pool().apply(super().f) or concurrent.futures.ProcessPoolExecutor().submit(super().f) in B.f where B is a subclass of A. This infinite recursion has crashed my laptop several times today, so it looks like it is a rather critical issue.

[maggyero]

@george Xie

There is an issue with the Python workaround multiprocessing.reduction.register(types.MethodType, my_reduce) that you proposed. If you then instantiate a multiprocessing.Manager() you get this exception:

> python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> import types
>>> def my_reduce(obj):
...     return (obj.__func__.__get__, (obj.__self__,))
...
>>> multiprocessing.reduction.register(types.MethodType, my_reduce)
>>> multiprocessing.Manager()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 56, in Manager
    m.start()
  File "C:\Program Files\Python37\lib\multiprocessing\managers.py", line 543, in start
    self._process.start()
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function BaseManager._run_server at 0x000001AEF8ED1D90>: it's not the same object as multiprocessing.managers.BaseManager._run_server
>>> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

[maggyero]

The previous _pickle.PicklingError seems to only affect Windows.