Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #90876

Closed
maggyero mannequin opened this issue Feb 11, 2022 · 2 comments
Labels
3.9 3.10 3.11 stdlib type-feature

Comments

@maggyero
Copy link

@maggyero maggyero mannequin commented Feb 11, 2022

BPO 46720
Nosy @maggyero
PRs
  • #31279
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2022-02-11.14:57:09.587>
    labels = ['type-feature', 'library', '3.9', '3.10', '3.11']
    title = 'Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems'
    updated_at = <Date 2022-02-11.22:25:00.201>
    user = 'https://github.com/maggyero'

    bugs.python.org fields:

    activity = <Date 2022-02-11.22:25:00.201>
    actor = 'maggyero'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2022-02-11.14:57:09.587>
    creator = 'maggyero'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 46720
    keywords = ['patch']
    message_count = 1.0
    messages = ['413073']
    nosy_count = 1.0
    nosy_names = ['maggyero']
    pr_nums = ['31279']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue46720'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @maggyero
    Copy link
    Author

    @maggyero maggyero mannequin commented Feb 11, 2022

    Any path-like object can be passed to multiprocessing.set_executable, i.e. objects with str, bytes, or os.PathLike type.

    For instance these work (tested on MacOS with all start methods: ‘spawn’, ‘fork’, and ‘forkserver’):

    • multiprocessing.set_executable(sys.executable) (str);
    • multiprocessing.set_executable(sys.executable.encode()) (bytes);
    • multiprocessing.set_executable(pathlib.Path(sys.executable)) (os.PathLike).

    This is because the ‘fork’ start method does not exec any program in the subprocess, the ‘spawn’ start method converts its path argument to bytes with os.fsencode before passing to _posixsubprocess.fork_exec, and the ‘forkserver’ start method spawns a server process (like with the ‘spawn’ start method) which then forks itself at each request (like the ‘fork’ start method):

            return _posixsubprocess.fork_exec(
                args, [os.fsencode(path)], True, passfds, None, None,
                -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
                False, False, None, None, None, -1, None)
    

    Linux (and other Unix-like systems) uses the same code than MacOS for the three start methods so it should work for it too.

    However I have not tested this on Windows which uses the function _winapi.CreateProcess for the ‘spawn’ start method (the only start method available on this OS) but I noticed that no conversion to str (not to bytes this time, since the function expects str) of the path argument with os.fsdecode (not os.fsencode this time) is performed before passing it to the function:

                    hp, ht, pid, tid = _winapi.CreateProcess(
                        python_exe, cmd,
                        None, None, False, 0, env, None, None)
    

    So on Windows only str path can be passed to multiprocessing.set_executable. This PR fixes this to be on a par with Unix-like systems which accept any path-like objects.

    @maggyero maggyero mannequin added 3.9 3.10 3.11 stdlib type-feature labels Feb 11, 2022
    @maggyero maggyero mannequin changed the title Add support of path-like objects to multiprocessing.set_executable for Windows to match Unix-like systems Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems Feb 11, 2022
    @maggyero maggyero mannequin changed the title Add support of path-like objects to multiprocessing.set_executable for Windows to match Unix-like systems Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems Feb 11, 2022
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @brettcannon
    Copy link

    @brettcannon brettcannon commented Apr 22, 2022

    Closed via #31279 .

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 3.10 3.11 stdlib type-feature
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant