Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .claude/settings.local.json
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,10 @@
"mcp__github__get_pull_request_review_comments",
"mcp__github__pull_request_read",
"Skill(run-tests)",
"Skill(lint)"
"Skill(lint)",
"Skill(find-cpython-usage)",
"Skill(compare-cpython-versions)"
],
"deny": []
}
}
}
171 changes: 171 additions & 0 deletions .claude/skills/compare-cpython-versions/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
---
name: compare-cpython-versions
description: >
Compare CPython source code between two Python versions to identify changes in
headers and structs. Use this when adding support for a new Python version to
understand what changed between versions.
allowed-tools:
- Bash
- Read
- Grep
- WebSearch
- TodoWrite
---

# Compare CPython Versions Skill

This skill helps compare CPython source code between two Python versions to identify
changes in headers, structs, and APIs that affect our codebase.

## When to Use This Skill

Use this skill when:
- Adding support for a new Python version
- Need to understand what changed between versions
- Investigating compatibility issues
- Have a list of headers/structs from `find-cpython-usage` skill

## Key Principles

1. **Compare systematically** - Focus on headers and structs identified in Step 1
2. **Use multiple methods** - Git diff, manual diff, or AI-assisted comparison
3. **Document changes** - Note all breaking changes and API modifications
4. **Check context** - Understand why changes were made (PEPs, GitHub issues)

## How This Skill Works

### Step 1: Prepare CPython Repository

```bash
# Create ~/dd directory if it doesn't exist
mkdir -p ~/dd

# Clone CPython repository if needed (to ~/dd/cpython)
if [ ! -d ~/dd/cpython ]; then
git clone https://github.com/python/cpython.git ~/dd/cpython
cd ~/dd/cpython
git fetch --tags
else
cd ~/dd/cpython
# Update existing repository
git fetch --tags
git fetch origin
fi
```

### Step 2: Compare Specific Headers

Using the list of headers from `find-cpython-usage`, compare each header between
the old version and new version. Replace `OLD_VERSION` and `NEW_VERSION` with the
actual version tags (e.g., `v3.13.0`, `v3.14.0`):

```bash
# Compare specific headers between versions
git diff OLD_VERSION NEW_VERSION -- Include/internal/pycore_frame.h
git diff OLD_VERSION NEW_VERSION -- Include/frameobject.h

# Compare all internal headers
git diff OLD_VERSION NEW_VERSION -- 'Include/internal/pycore*.h'

# Compare specific struct definitions
git diff OLD_VERSION NEW_VERSION -- Include/internal/pycore_frame.h | grep -A 20 "struct _PyInterpreterFrame"
```


### Step 3: Identify Changes

For each header/struct, look for:

**Struct Changes:**
- Field additions
- Field removals
- Field type changes
- Field reordering
- Struct moves to different headers

**API Changes:**
- Removed functions/structures
- New functions/structures
- Changed function signatures
- Deprecated APIs

**Header Changes:**
- Headers moved to different locations
- Headers split or merged
- New headers introduced

### Step 4: Analyze Impact

For each change identified:

1. **Understand the change:**
- Why was it changed? (Check Python's What's New, PEPs, or GitHub issues)
- Is it a breaking change or backward compatible?
- What's the replacement API?
- **Find the specific commit(s) that introduced the change:**

```bash
# Find commits that modified a specific file between versions
git log OLD_VERSION..NEW_VERSION -- Include/internal/pycore_frame.h

# Find commits that mention a specific struct or function
git log OLD_VERSION..NEW_VERSION --all --grep="_PyInterpreterFrame" -- Include/

# Show the commit that introduced a specific change
git log -p OLD_VERSION..NEW_VERSION -S "struct _PyInterpreterFrame" -- Include/
```

- **Find related GitHub issues:**
- Check commit messages for issue references (e.g., `gh-123923`, `#123923`)
- Search CPython GitHub issues: `https://github.com/python/cpython/issues`
- Look for "What's New" documentation: `https://docs.python.org/3/whatsnew/`
- Check PEPs if the change is part of a larger feature

2. **Assess impact:**
- Which files in our codebase are affected?
- What functionality might break?
- Are there alternative approaches?

3. **Document findings:**
- Create a summary document of key changes
- Note any breaking changes
- List files that need updates

### Step 5: Use AI Tools (Optional)

You can use AI coding assistants to help analyze differences by:
- Providing header file contents from both versions
- Asking about specific struct changes
- Understanding migration paths

## Common Change Patterns

When comparing versions, look for these types of changes (examples):

**Struct Field Changes:**
- Field type changes (e.g., pointer types → tagged pointer types)
- Field renamed
- Field removed and replaced with different mechanism
- Field reordering

**Header Moves:**
- Internal headers moved to new locations
- Structs moved between headers
- Headers split or merged

**API Deprecations:**
- Internal functions removed
- Public API replacements available
- Function signature changes

## Output Format

After running this skill, you should have:
1. A list of all changed headers
2. A list of all changed structs with details
3. Impact assessment for each change
4. Files in our codebase that need updates

## Related

- **find-cpython-usage skill**: Use to identify what to compare
167 changes: 167 additions & 0 deletions .claude/skills/find-cpython-usage/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
---
name: find-cpython-usage
description: >
Find all CPython internal headers and structs used in the codebase, particularly
for profiling functionality. Use this when adding support for a new Python version
to identify what CPython internals we depend on.
allowed-tools:
- Bash
- Read
- Grep
- Glob
- TodoWrite
---

# Find CPython Internal Usage Skill

This skill helps identify all CPython internal headers and structures used in the
codebase, which is essential when adding support for new Python versions.

## When to Use This Skill

Use this skill when:
- Adding support for a new Python version
- Investigating CPython API dependencies
- Understanding what internal APIs the profiler uses
- Preparing to compare CPython versions

## Key Principles

1. **Focus on internal headers** - These are most likely to change between versions
2. **Check all native extensions** - CPython internals are used in profiling, AppSec, and internal modules
3. **Look for struct field access** - Direct field access is version-sensitive
4. **Document findings** - Keep track of what you find for comparison

## How This Skill Works

### Step 1: Find CPython Header Includes

Search for CPython header includes across all C/C++/Cython files:

```bash
# Find all CPython internal header includes
grep -r "include.*internal/pycore" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find all CPython cpython header includes
grep -r "include.*cpython" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find all Python.h includes (indicates CPython API usage)
grep -r "#include.*Python\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find frameobject.h includes (common CPython API)
grep -r "#include.*frameobject\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find PyO3 FFI usage in Rust (Rust extensions may use CPython internals)
grep -r "pyo3_ffi::\|pyo3::ffi::" src/native/ --include="*.rs" || true
```

**Note:** These patterns search across all native extension files (`.c`, `.cpp`, `.h`, `.hpp`, `.pyx`, `.rs`)
regardless of their location in the codebase. Check `setup.py` to see which extensions are built.
Rust extensions use PyO3 which may access CPython internals through the `pyo3_ffi` module.

### Step 2: Find Struct Field Access

Search for direct struct field access and struct definitions across all native files:

```bash
# Find struct field accesses (arrow operator)
grep -r "->f_\|->[a-z_]*\." ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp"

# Find struct definitions
grep -r "struct.*Py" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp"

# Find common CPython struct usage
grep -r "PyFrameObject\|PyThreadState\|_PyInterpreterFrame" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
grep -r "PyFrameObject\|PyThreadState\|PyInterpreterState" src/native/ --include="*.rs" || true

# Find PyCodeObject usage
grep -r "PyCodeObject" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
grep -r "PyCodeObject" src/native/ --include="*.rs" || true
```

**Look for patterns like:**
- Frame structure access (`PyFrameObject`, `_PyInterpreterFrame`)
- Thread state access (`PyThreadState`)
- Code object access (`PyCodeObject`)
- Generator/coroutine structures
- Asyncio task structures

### Step 3: Identify Common Structures

Common CPython structures we typically access:

**Frame structures:**
- `PyFrameObject` / `struct _frame`
- `_PyInterpreterFrame`

**State structures:**
- `PyThreadState`
- `PyInterpreterState`
- `_PyRuntimeState`

**Code structures:**
- `PyCodeObject`

**Generator structures:**
- `PyGenObject`
- `PyAsyncGenASend`

**Asyncio structures:**
- `FutureObj`
- `TaskObj`

### Step 4: Document Findings

Create a list of:
- All headers that are included
- All structs that are accessed
- All struct fields that are used directly

This will be used in the next step to compare against the new Python version.

## Native Extensions Using CPython APIs

To find all native extensions that may use CPython APIs, check `setup.py`:

```bash
# View all native extensions defined in setup.py
grep -A 5 "Extension\|CMakeExtension\|Cython.Distutils.Extension\|RustExtension" setup.py
```

The `setup.py` file defines all native extensions (C, C++, CMake, Cython, and Rust) that
are built for the project. Not all extensions use CPython internals - focus on those
that access frame objects, thread state, or internal structures when searching for
CPython API usage.

**Note:** Rust extensions use PyO3 bindings which may access CPython internals through
`pyo3_ffi` module. Search Rust source files (`.rs`) for CPython API usage as well.

## Common Headers to Look For

The grep commands above will identify which CPython headers are actually used in the codebase.
Common patterns include:

**Public Headers:**
- Headers matching `*.h` in `Include/` directory (e.g., `frameobject.h`, `unicodeobject.h`)
- Headers in `Include/cpython/` directory (e.g., `cpython/genobject.h`)

**Internal Headers (require `Py_BUILD_CORE`):**
- Headers matching `internal/pycore*.h` pattern
- These are most likely to change between Python versions

Focus on headers that are actually found by the grep commands rather than maintaining
a hardcoded list, as the headers used may change over time.

## Output Format

After running this skill, you should have:
1. A list of all CPython headers included in the codebase
2. A list of all CPython structs accessed
3. A list of struct fields accessed directly
4. Files that use each header/struct

This information can then be used with the `compare-cpython-versions` skill to identify what changed.

## Related

- **compare-cpython-versions skill**: Use findings from this skill to compare versions
22 changes: 22 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,28 @@ This project has custom skills that provide specialized workflows. **Always chec

**Usage:** Use the Skill tool with command "lint"

### find-cpython-usage

**Use whenever:** Adding support for a new Python version or investigating CPython API dependencies.

**Purpose:** Identifies all CPython internal headers and structures used in the codebase:
- Searches for CPython header includes
- Finds struct field accesses
- Documents CPython internals we depend on

**Usage:** Use the Skill tool with command "find-cpython-usage"

### compare-cpython-versions

**Use whenever:** Comparing CPython source code between two Python versions to identify changes.

**Purpose:** Compares CPython headers and structs between versions:
- Uses git diff or manual diff to compare versions
- Identifies breaking changes and API modifications
- Assesses impact on our codebase

**Usage:** Use the Skill tool with command "compare-cpython-versions"

---

<!-- Add more skills below as they are created -->
Expand Down
Loading