Skip to content

Commit 8969ce6

Browse files
committed
Add --children flag to profile child processes
The ChildProcessMonitor class runs a background thread that polls for new child processes spawned by the target. When it finds a new Python process, it launches a separate profiler subprocess with the same sampling options. Each child profiler writes to its own output file with the child's PID appended to the filename pattern. Detection uses a fast path on Linux (checking /proc/{pid}/exe for "python" in the name) before falling back to the full RemoteUnwinder probe. Non-Python children are silently skipped. There's a limit of 100 concurrent child profilers to avoid runaway resource usage if the target forks heavily. The --children flag is incompatible with --live mode since the curses interface can't handle multiple profiler outputs simultaneously.
1 parent 7dbf221 commit 8969ce6

File tree

3 files changed

+510
-42
lines changed

3 files changed

+510
-42
lines changed

Doc/library/profiling.sampling.rst

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,6 +310,8 @@ The default configuration works well for most use cases:
310310
- Wall-clock mode (all samples recorded)
311311
* - ``--realtime-stats``
312312
- No live statistics display during profiling
313+
* - ``--children``
314+
- Profile only the target process (no child process monitoring)
313315

314316

315317
Sampling interval and duration
@@ -442,6 +444,79 @@ working correctly and that sufficient samples are being collected. See
442444
:ref:`sampling-efficiency` for details on interpreting these metrics.
443445

444446

447+
Child process profiling
448+
-----------------------
449+
450+
The :option:`--children` option enables automatic profiling of child processes
451+
spawned by the target::
452+
453+
python -m profiling.sampling run --children script.py
454+
python -m profiling.sampling attach --children 12345
455+
456+
When enabled, the profiler monitors the target process for child process
457+
creation. When a new Python child process is detected, a separate profiler
458+
instance is automatically spawned to profile it. This is useful for
459+
applications that use :mod:`multiprocessing`, :mod:`subprocess`,
460+
:mod:`concurrent.futures` with :class:`~concurrent.futures.ProcessPoolExecutor`,
461+
or other process spawning mechanisms.
462+
463+
.. code-block:: python
464+
465+
# worker_pool.py
466+
from concurrent.futures import ProcessPoolExecutor
467+
import math
468+
469+
def compute_factorial(n):
470+
total = 0
471+
for i in range(50):
472+
total += math.factorial(n)
473+
return total
474+
475+
if __name__ == "__main__":
476+
numbers = [5000 + i * 100 for i in range(50)]
477+
with ProcessPoolExecutor(max_workers=4) as executor:
478+
results = list(executor.map(compute_factorial, numbers))
479+
print(f"Computed {len(results)} factorials")
480+
481+
::
482+
483+
python -m profiling.sampling run --children --flamegraph worker_pool.py
484+
485+
This produces separate flame graphs for the main process and each worker
486+
process: ``flamegraph.<main_pid>.html``, ``flamegraph.<worker1_pid>.html``,
487+
and so on.
488+
489+
Each child process receives its own output file. The filename is derived from
490+
the specified output path (or the default) with the child's process ID
491+
appended:
492+
493+
- If you specify ``-o profile.html``, children produce ``profile_12345.html``,
494+
``profile_12346.html``, and so on
495+
- With default output, children produce files like ``flamegraph.12345.html``
496+
or directories like ``heatmap_12345``
497+
- For pstats format (which defaults to stdout), children produce files like
498+
``profile.12345.pstats``
499+
500+
The child profilers inherit most sampling options from the parent (interval,
501+
duration, thread selection, native frames, GC frames, async-aware mode, and
502+
output format). All Python descendant processes are profiled recursively,
503+
including grandchildren and further descendants.
504+
505+
Child process detection works by periodically scanning for new descendants of
506+
the target process and checking whether each new process is a Python process.
507+
On Linux, this uses a fast check of the executable name followed by a full
508+
probe of the process memory if needed. Non-Python child processes (such as
509+
shell commands or external tools) are ignored.
510+
511+
There is a limit of 100 concurrent child profilers to prevent resource
512+
exhaustion in programs that spawn many processes. If this limit is reached,
513+
additional child processes are not profiled and a warning is printed.
514+
515+
The :option:`--children` option is incompatible with :option:`--live` mode
516+
because live mode uses an interactive terminal interface that cannot
517+
accommodate multiple concurrent profiler displays.
518+
519+
445520
.. _sampling-efficiency:
446521

447522
Sampling efficiency
@@ -1128,6 +1203,11 @@ Sampling options
11281203
Compatible with ``--live``, ``--flamegraph``, ``--heatmap``, and ``--gecko``
11291204
formats only.
11301205

1206+
.. option:: --children
1207+
1208+
Also profile child processes. Each child process gets its own profiler
1209+
instance and output file. Incompatible with ``--live``.
1210+
11311211

11321212
Mode options
11331213
------------

0 commit comments

Comments
 (0)