gh-138122: Add --subprocesses flag to profile child processes in tachyon #142636
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The _GNU_SOURCE macro must be defined before any system headers are included to enable GNU extensions like process_vm_readv. Moving it before the extern "C" block ensures it takes effect. The internal Python headers are also changed from angle brackets to quotes since they're local to the project. On macOS, the TARGET_OS_OSX macro may not be defined by older SDKs, so we now include TargetConditionals.h explicitly and provide a fallback definition when needed.
This implements platform-specific child process enumeration for use by
the profiler. On Linux it parses /proc/{pid}/stat to build a parent-
child map and then walks the tree from the target PID. On macOS it uses
proc_listchildpids() when available, falling back to scanning all
processes with proc_pidinfo(). On Windows it uses CreateToolhelp32Snapshot
with TH32CS_SNAPPROCESS to iterate through all processes.
The function returns a Python list of PIDs representing all descendants
of the given process. The recursive parameter controls whether only
direct children or all descendants are returned. This is the building
block needed for the --children flag in the sampling profiler CLI.
The ChildProcessMonitor class runs a background thread that polls for
new child processes spawned by the target. When it finds a new Python
process, it launches a separate profiler subprocess with the same
sampling options. Each child profiler writes to its own output file
with the child's PID appended to the filename pattern.
Detection uses a fast path on Linux (checking /proc/{pid}/exe for
"python" in the name) before falling back to the full RemoteUnwinder
probe. Non-Python children are silently skipped. There's a limit of
100 concurrent child profilers to avoid runaway resource usage if the
target forks heavily.
The --children flag is incompatible with --live mode since the curses
interface can't handle multiple profiler outputs simultaneously.
The test suite covers the get_child_pids C function with both recursive and non-recursive enumeration, the is_python_process detection helper, the ChildProcessMonitor lifecycle, and end-to-end CLI tests with --children for both attach and run modes. Tests use short-lived subprocesses with controlled lifecycles and polling-based synchronization rather than fixed sleeps to keep runtime reasonable. Resource cleanup is handled through reap_children and explicit process termination in finally blocks.
pablogsal
force-pushed
the
tachyon-subprocesses-atomic
branch
from
2d7e614 to
e6ca9f9
Compare
pablogsal
requested review from
StanFromIreland,
ivonastojanovic and
savannahostrowski
Member
Author
|
CC: @lkollar |
pablogsal
force-pushed
the
tachyon-subprocesses-atomic
branch
from
ebdb847 to
bc5dc46
Compare
pablogsal
changed the title
Member
Author
|
I know this looks like "Oh no another 2k line PR from Pablo 😆 " (and it kind of is) but a lot of this is autogenerated files, tests and docs :) |
pablogsal
requested review from
a team,
AA-Turner,
emmatyping and
erlend-aasland
as code owners
savannahostrowski
requested changes
Dec 12, 2025
|
When you're done making the requested changes, leave the comment: |
pablogsal
force-pushed
the
tachyon-subprocesses-atomic
branch
from
6647f4a to
9ffd19d
Compare
pablogsal
force-pushed
the
tachyon-subprocesses-atomic
branch
from
9ffd19d to
42d5119
Compare
Member
Author
|
I have made the requested changes; please review again |
|
Thanks for making the requested changes! @savannahostrowski: please review the changes made to this pull request. |
savannahostrowski
approved these changes
Dec 14, 2025
Member
savannahostrowski
left a comment
savannahostrowski
left a comment
There was a problem hiding this comment.
One small comment but otherwise LGTM!
Co-authored-by: Savannah Ostrowski <savannah@python.org>
pablogsal
added
the
🔨 test-with-buildbots
|
🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 50b7c89 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F142636%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
bedevere-bot
removed
the
🔨 test-with-buildbots
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the ability to automatically profile child processes spawned by the target when using the sampling profiler. When the
--subprocessesflag is passed, a background monitor thread polls for new descendants of the target process and spawns separate profiler instances for each Python child it discovers. Each child profiler inherits the sampling options from the parent (interval, duration, thread selection, native frames, async-aware mode, output format) and writes to its own output file with the child's PID appended to the filename.There is a limit of 100 concurrent child profilers to prevent resource exhaustion when profiling applications that fork heavily. The
--subprocessesflag is incompatible with--livemode since the curses interface cannot accommodate multiple concurrent profiler displays. This is useful for profiling applications that usemultiprocessing,ProcessPoolExecutor, or other subprocess-based parallelism.