diff --git a/.claude/settings.local.json b/.claude/settings.local.json index c43307ad5a3..0dfb25367cd 100644 --- a/.claude/settings.local.json +++ b/.claude/settings.local.json @@ -35,8 +35,10 @@ "mcp__github__get_pull_request_review_comments", "mcp__github__pull_request_read", "Skill(run-tests)", - "Skill(lint)" + "Skill(lint)", + "Skill(find-cpython-usage)", + "Skill(compare-cpython-versions)" ], "deny": [] } -} \ No newline at end of file +} diff --git a/.claude/skills/compare-cpython-versions/SKILL.md b/.claude/skills/compare-cpython-versions/SKILL.md new file mode 100644 index 00000000000..dd048b4e1b5 --- /dev/null +++ b/.claude/skills/compare-cpython-versions/SKILL.md @@ -0,0 +1,171 @@ +--- +name: compare-cpython-versions +description: > + Compare CPython source code between two Python versions to identify changes in + headers and structs. Use this when adding support for a new Python version to + understand what changed between versions. +allowed-tools: + - Bash + - Read + - Grep + - WebSearch + - TodoWrite +--- + +# Compare CPython Versions Skill + +This skill helps compare CPython source code between two Python versions to identify +changes in headers, structs, and APIs that affect our codebase. + +## When to Use This Skill + +Use this skill when: +- Adding support for a new Python version +- Need to understand what changed between versions +- Investigating compatibility issues +- Have a list of headers/structs from `find-cpython-usage` skill + +## Key Principles + +1. **Compare systematically** - Focus on headers and structs identified in Step 1 +2. **Use multiple methods** - Git diff, manual diff, or AI-assisted comparison +3. **Document changes** - Note all breaking changes and API modifications +4. **Check context** - Understand why changes were made (PEPs, GitHub issues) + +## How This Skill Works + +### Step 1: Prepare CPython Repository + +```bash +# Create ~/dd directory if it doesn't exist +mkdir -p ~/dd + +# Clone CPython repository if needed (to ~/dd/cpython) +if [ ! -d ~/dd/cpython ]; then + git clone https://github.com/python/cpython.git ~/dd/cpython + cd ~/dd/cpython + git fetch --tags +else + cd ~/dd/cpython + # Update existing repository + git fetch --tags + git fetch origin +fi +``` + +### Step 2: Compare Specific Headers + +Using the list of headers from `find-cpython-usage`, compare each header between +the old version and new version. Replace `OLD_VERSION` and `NEW_VERSION` with the +actual version tags (e.g., `v3.13.0`, `v3.14.0`): + +```bash +# Compare specific headers between versions +git diff OLD_VERSION NEW_VERSION -- Include/internal/pycore_frame.h +git diff OLD_VERSION NEW_VERSION -- Include/frameobject.h + +# Compare all internal headers +git diff OLD_VERSION NEW_VERSION -- 'Include/internal/pycore*.h' + +# Compare specific struct definitions +git diff OLD_VERSION NEW_VERSION -- Include/internal/pycore_frame.h | grep -A 20 "struct _PyInterpreterFrame" +``` + + +### Step 3: Identify Changes + +For each header/struct, look for: + +**Struct Changes:** +- Field additions +- Field removals +- Field type changes +- Field reordering +- Struct moves to different headers + +**API Changes:** +- Removed functions/structures +- New functions/structures +- Changed function signatures +- Deprecated APIs + +**Header Changes:** +- Headers moved to different locations +- Headers split or merged +- New headers introduced + +### Step 4: Analyze Impact + +For each change identified: + +1. **Understand the change:** + - Why was it changed? (Check Python's What's New, PEPs, or GitHub issues) + - Is it a breaking change or backward compatible? + - What's the replacement API? + - **Find the specific commit(s) that introduced the change:** + + ```bash + # Find commits that modified a specific file between versions + git log OLD_VERSION..NEW_VERSION -- Include/internal/pycore_frame.h + + # Find commits that mention a specific struct or function + git log OLD_VERSION..NEW_VERSION --all --grep="_PyInterpreterFrame" -- Include/ + + # Show the commit that introduced a specific change + git log -p OLD_VERSION..NEW_VERSION -S "struct _PyInterpreterFrame" -- Include/ + ``` + + - **Find related GitHub issues:** + - Check commit messages for issue references (e.g., `gh-123923`, `#123923`) + - Search CPython GitHub issues: `https://github.com/python/cpython/issues` + - Look for "What's New" documentation: `https://docs.python.org/3/whatsnew/` + - Check PEPs if the change is part of a larger feature + +2. **Assess impact:** + - Which files in our codebase are affected? + - What functionality might break? + - Are there alternative approaches? + +3. **Document findings:** + - Create a summary document of key changes + - Note any breaking changes + - List files that need updates + +### Step 5: Use AI Tools (Optional) + +You can use AI coding assistants to help analyze differences by: +- Providing header file contents from both versions +- Asking about specific struct changes +- Understanding migration paths + +## Common Change Patterns + +When comparing versions, look for these types of changes (examples): + +**Struct Field Changes:** +- Field type changes (e.g., pointer types → tagged pointer types) +- Field renamed +- Field removed and replaced with different mechanism +- Field reordering + +**Header Moves:** +- Internal headers moved to new locations +- Structs moved between headers +- Headers split or merged + +**API Deprecations:** +- Internal functions removed +- Public API replacements available +- Function signature changes + +## Output Format + +After running this skill, you should have: +1. A list of all changed headers +2. A list of all changed structs with details +3. Impact assessment for each change +4. Files in our codebase that need updates + +## Related + +- **find-cpython-usage skill**: Use to identify what to compare diff --git a/.claude/skills/find-cpython-usage/SKILL.md b/.claude/skills/find-cpython-usage/SKILL.md new file mode 100644 index 00000000000..71ccaf1bb25 --- /dev/null +++ b/.claude/skills/find-cpython-usage/SKILL.md @@ -0,0 +1,167 @@ +--- +name: find-cpython-usage +description: > + Find all CPython internal headers and structs used in the codebase, particularly + for profiling functionality. Use this when adding support for a new Python version + to identify what CPython internals we depend on. +allowed-tools: + - Bash + - Read + - Grep + - Glob + - TodoWrite +--- + +# Find CPython Internal Usage Skill + +This skill helps identify all CPython internal headers and structures used in the +codebase, which is essential when adding support for new Python versions. + +## When to Use This Skill + +Use this skill when: +- Adding support for a new Python version +- Investigating CPython API dependencies +- Understanding what internal APIs the profiler uses +- Preparing to compare CPython versions + +## Key Principles + +1. **Focus on internal headers** - These are most likely to change between versions +2. **Check all native extensions** - CPython internals are used in profiling, AppSec, and internal modules +3. **Look for struct field access** - Direct field access is version-sensitive +4. **Document findings** - Keep track of what you find for comparison + +## How This Skill Works + +### Step 1: Find CPython Header Includes + +Search for CPython header includes across all C/C++/Cython files: + +```bash +# Find all CPython internal header includes +grep -r "include.*internal/pycore" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx" + +# Find all CPython cpython header includes +grep -r "include.*cpython" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx" + +# Find all Python.h includes (indicates CPython API usage) +grep -r "#include.*Python\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx" + +# Find frameobject.h includes (common CPython API) +grep -r "#include.*frameobject\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx" + +# Find PyO3 FFI usage in Rust (Rust extensions may use CPython internals) +grep -r "pyo3_ffi::\|pyo3::ffi::" src/native/ --include="*.rs" || true +``` + +**Note:** These patterns search across all native extension files (`.c`, `.cpp`, `.h`, `.hpp`, `.pyx`, `.rs`) +regardless of their location in the codebase. Check `setup.py` to see which extensions are built. +Rust extensions use PyO3 which may access CPython internals through the `pyo3_ffi` module. + +### Step 2: Find Struct Field Access + +Search for direct struct field access and struct definitions across all native files: + +```bash +# Find struct field accesses (arrow operator) +grep -r "->f_\|->[a-z_]*\." ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" + +# Find struct definitions +grep -r "struct.*Py" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" + +# Find common CPython struct usage +grep -r "PyFrameObject\|PyThreadState\|_PyInterpreterFrame" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx" +grep -r "PyFrameObject\|PyThreadState\|PyInterpreterState" src/native/ --include="*.rs" || true + +# Find PyCodeObject usage +grep -r "PyCodeObject" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx" +grep -r "PyCodeObject" src/native/ --include="*.rs" || true +``` + +**Look for patterns like:** +- Frame structure access (`PyFrameObject`, `_PyInterpreterFrame`) +- Thread state access (`PyThreadState`) +- Code object access (`PyCodeObject`) +- Generator/coroutine structures +- Asyncio task structures + +### Step 3: Identify Common Structures + +Common CPython structures we typically access: + +**Frame structures:** +- `PyFrameObject` / `struct _frame` +- `_PyInterpreterFrame` + +**State structures:** +- `PyThreadState` +- `PyInterpreterState` +- `_PyRuntimeState` + +**Code structures:** +- `PyCodeObject` + +**Generator structures:** +- `PyGenObject` +- `PyAsyncGenASend` + +**Asyncio structures:** +- `FutureObj` +- `TaskObj` + +### Step 4: Document Findings + +Create a list of: +- All headers that are included +- All structs that are accessed +- All struct fields that are used directly + +This will be used in the next step to compare against the new Python version. + +## Native Extensions Using CPython APIs + +To find all native extensions that may use CPython APIs, check `setup.py`: + +```bash +# View all native extensions defined in setup.py +grep -A 5 "Extension\|CMakeExtension\|Cython.Distutils.Extension\|RustExtension" setup.py +``` + +The `setup.py` file defines all native extensions (C, C++, CMake, Cython, and Rust) that +are built for the project. Not all extensions use CPython internals - focus on those +that access frame objects, thread state, or internal structures when searching for +CPython API usage. + +**Note:** Rust extensions use PyO3 bindings which may access CPython internals through +`pyo3_ffi` module. Search Rust source files (`.rs`) for CPython API usage as well. + +## Common Headers to Look For + +The grep commands above will identify which CPython headers are actually used in the codebase. +Common patterns include: + +**Public Headers:** +- Headers matching `*.h` in `Include/` directory (e.g., `frameobject.h`, `unicodeobject.h`) +- Headers in `Include/cpython/` directory (e.g., `cpython/genobject.h`) + +**Internal Headers (require `Py_BUILD_CORE`):** +- Headers matching `internal/pycore*.h` pattern +- These are most likely to change between Python versions + +Focus on headers that are actually found by the grep commands rather than maintaining +a hardcoded list, as the headers used may change over time. + +## Output Format + +After running this skill, you should have: +1. A list of all CPython headers included in the codebase +2. A list of all CPython structs accessed +3. A list of struct fields accessed directly +4. Files that use each header/struct + +This information can then be used with the `compare-cpython-versions` skill to identify what changed. + +## Related + +- **compare-cpython-versions skill**: Use findings from this skill to compare versions diff --git a/AGENTS.md b/AGENTS.md index cad07d8a619..c6535e24a2e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,28 @@ This project has custom skills that provide specialized workflows. **Always chec **Usage:** Use the Skill tool with command "lint" +### find-cpython-usage + +**Use whenever:** Adding support for a new Python version or investigating CPython API dependencies. + +**Purpose:** Identifies all CPython internal headers and structures used in the codebase: +- Searches for CPython header includes +- Finds struct field accesses +- Documents CPython internals we depend on + +**Usage:** Use the Skill tool with command "find-cpython-usage" + +### compare-cpython-versions + +**Use whenever:** Comparing CPython source code between two Python versions to identify changes. + +**Purpose:** Compares CPython headers and structs between versions: +- Uses git diff or manual diff to compare versions +- Identifies breaking changes and API modifications +- Assesses impact on our codebase + +**Usage:** Use the Skill tool with command "compare-cpython-versions" + ---