Skip to content

gh-85222: Document the side effect in multiprocessing #136426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 72 additions & 14 deletions Doc/library/multiprocessing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,21 @@
The :mod:`multiprocessing` package mostly replicates the API of the
:mod:`threading` module.

.. _global-start-method:

Global start method
^^^^^^^^^^^^^^^^^^^

Python supports several ways to create and initialize a process.
The global start method sets the default mechanism for creating a process.

Several multiprocessing functions and methods that may also instantiate
certain objects will implicitly set the global start method to the system's default,
if it hasn’t been set already. The global start method can only be set once.
If you need to change the start method from the system default, you must
proactively set the global start method before calling functions or methods,
or creating these objects.


:class:`Process` and exceptions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -867,6 +882,9 @@
locks/semaphores. When a process first puts an item on the queue a feeder
thread is started which transfers objects from a buffer into the pipe.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

The usual :exc:`queue.Empty` and :exc:`queue.Full` exceptions from the
standard library's :mod:`queue` module are raised to signal timeouts.

Expand Down Expand Up @@ -977,6 +995,9 @@

It is a simplified :class:`Queue` type, very close to a locked :class:`Pipe`.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

.. method:: close()

Close the queue: release internal resources.
Expand Down Expand Up @@ -1007,6 +1028,9 @@
:class:`JoinableQueue`, a :class:`Queue` subclass, is a queue which
additionally has :meth:`task_done` and :meth:`join` methods.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

.. method:: task_done()

Indicate that a formerly enqueued task is complete. Used by queue
Expand Down Expand Up @@ -1119,8 +1143,8 @@
:mod:`multiprocessing` module.

If *method* is ``None`` then the default context is returned. Note that if
the global start method has not been set, this will set it to the
default method.
the global start method has not been set, this will set it to the system default
See :ref:`global-start-method` for more details.
Otherwise *method* should be ``'fork'``, ``'spawn'``,
``'forkserver'``. :exc:`ValueError` is raised if the specified
start method is not available. See :ref:`multiprocessing-start-methods`.
Expand All @@ -1131,10 +1155,9 @@

Return the name of start method used for starting processes.

If the global start method has not been set and *allow_none* is
``False``, then the start method is set to the default and the name
is returned. If the start method has not been set and *allow_none* is
``True`` then ``None`` is returned.
If the global start method is not set and *allow_none* is ``False``, the global start
method is set to the default, and its name is returned. See
:ref:`global-start-method` for more details.

The return value can be ``'fork'``, ``'spawn'``, ``'forkserver'``
or ``None``. See :ref:`multiprocessing-start-methods`.
Expand Down Expand Up @@ -1361,13 +1384,19 @@

A barrier object: a clone of :class:`threading.Barrier`.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

.. versionadded:: 3.3

.. class:: BoundedSemaphore([value])

A bounded semaphore object: a close analog of
:class:`threading.BoundedSemaphore`.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

A solitary difference from its close analog exists: its ``acquire`` method's
first argument is named *block*, as is consistent with :meth:`Lock.acquire`.

Expand All @@ -1388,13 +1417,18 @@
If *lock* is specified then it should be a :class:`Lock` or :class:`RLock`
object from :mod:`multiprocessing`.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

.. versionchanged:: 3.3
The :meth:`~threading.Condition.wait_for` method was added.

.. class:: Event()

A clone of :class:`threading.Event`.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

.. class:: Lock()

Expand All @@ -1410,6 +1444,9 @@
instance of ``multiprocessing.synchronize.Lock`` initialized with a
default context.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

:class:`Lock` supports the :term:`context manager` protocol and thus may be
used in :keyword:`with` statements.

Expand Down Expand Up @@ -1467,6 +1504,9 @@
instance of ``multiprocessing.synchronize.RLock`` initialized with a
default context.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

:class:`RLock` supports the :term:`context manager` protocol and thus may be
used in :keyword:`with` statements.

Expand Down Expand Up @@ -1526,6 +1566,9 @@

A semaphore object: a close analog of :class:`threading.Semaphore`.

Instantiating this class may set the global start method. See
:ref:`global-start-method` for more details.

A solitary difference from its close analog exists: its ``acquire`` method's
first argument is named *block*, as is consistent with :meth:`Lock.acquire`.

Expand Down Expand Up @@ -1660,7 +1703,7 @@
attributes which allow one to use it to store and retrieve strings -- see
documentation for :mod:`ctypes`.

.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True)
.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True, ctx=None)

The same as :func:`RawArray` except that depending on the value of *lock* a
process-safe synchronization wrapper may be returned instead of a raw ctypes
Expand All @@ -1674,9 +1717,13 @@
automatically protected by a lock, so it will not necessarily be
"process-safe".

Note that *lock* is a keyword-only argument.
*ctx* is a context object, or ``None`` (use the current context). If ``None``,
calling this may set the global start method. See
:ref:`global-start-method` for more details.

.. function:: Value(typecode_or_type, *args, lock=True)
Note that *lock* and *ctx* are keyword-only parameters.

.. function:: Value(typecode_or_type, *args, lock=True, ctx=None)

The same as :func:`RawValue` except that depending on the value of *lock* a
process-safe synchronization wrapper may be returned instead of a raw ctypes
Expand All @@ -1689,20 +1736,28 @@
automatically protected by a lock, so it will not necessarily be
"process-safe".

Note that *lock* is a keyword-only argument.
*ctx* is a context object, or ``None`` (use the current context). If ``None``,
calling this may set the global start method. See
:ref:`global-start-method` for more details.

Note that *lock* and *ctx* are keyword-only parameters.

.. function:: copy(obj)

Return a ctypes object allocated from shared memory which is a copy of the
ctypes object *obj*.

.. function:: synchronized(obj[, lock])
.. function:: synchronized(obj, lock=None, ctx=None)

Return a process-safe wrapper object for a ctypes object which uses *lock* to
synchronize access. If *lock* is ``None`` (the default) then a
:class:`multiprocessing.RLock` object is created automatically.

*ctx* is a context object, or ``None`` (use the current context). If ``None``,
calling this may set the global start method. See
:ref:`global-start-method` for more details.

A synchronized wrapper will have two methods in addition to those of the

Check warning on line 1760 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

py:meth reference target not found: get_lock [ref.meth]

Check warning on line 1760 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

py:meth reference target not found: get_obj [ref.meth]
object it wraps: :meth:`get_obj` returns the wrapped object and
:meth:`get_lock` returns the lock object used for synchronization.

Expand Down Expand Up @@ -1819,8 +1874,9 @@
*serializer* must be ``'pickle'`` (use :mod:`pickle` serialization) or
``'xmlrpclib'`` (use :mod:`xmlrpc.client` serialization).

*ctx* is a context object, or ``None`` (use the current context). See the
:func:`get_context` function.
*ctx* is a context object, or ``None`` (use the current context). If ``None``,
calling this may set the global start method. See
:ref:`global-start-method` for more details.

*shutdown_timeout* is a timeout in seconds used to wait until the process
used by the manager completes in the :meth:`shutdown` method. If the
Expand Down Expand Up @@ -2309,11 +2365,13 @@
unused resources to be freed. The default *maxtasksperchild* is ``None``, which
means worker processes will live as long as the pool.

*context* can be used to specify the context used for starting

Check warning on line 2368 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

py:func reference target not found: get_context [ref.func]

Check warning on line 2368 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

py:func reference target not found: multiprocessing.Pool [ref.func]
the worker processes. Usually a pool is created using the
function :func:`multiprocessing.Pool` or the :meth:`Pool` method
of a context object. In both cases *context* is set
appropriately.
appropriately. If ``None``, calling this function will have the side effect
of setting the current global start method if it has not been set already.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
of setting the current global start method if it has not been set already.
of setting the current global start method to the
system default if it has not been set already.

More small suggestions to improve reading flow

Up to you if you wanna take this one

See the :func:`get_context` function.

Note that the methods of the pool object should only be called by
the process which created the pool.
Expand Down
Loading