Commit c94cf91
[Server-Side Planning] Integrate DeltaCatalog with ServerSidePlannedTable and add E2E tests (#9)
* [Server-Side Planning] Integrate DeltaCatalog with ServerSidePlannedTable and add E2E tests
Implements DeltaCatalog integration to use ServerSidePlannedTable when Unity Catalog
tables lack credentials, with comprehensive end-to-end testing and improvements.
Key changes:
- Add loadTable() logic in DeltaCatalog to detect UC tables without credentials
- Implement hasCredentials() to check for credential properties in table metadata
- Add ENABLE_SERVER_SIDE_PLANNING config flag for testing
- Add comprehensive integration tests with reflection-based credential testing
- Add Spark source code references for Identifier namespace structure
- Improve test suite by removing redundant aggregation test
- Revert verbose documentation comments in ServerSidePlannedTable
Test coverage:
- E2E: Full stack integration with DeltaCatalog
- E2E: Verify normal path unchanged when feature disabled
- loadTable() decision logic with ENABLE_SERVER_SIDE_PLANNING config
(tests 3 scenarios including UC without credentials via reflection)
See Spark's LookupCatalog, CatalogAndIdentifier and ResolveSessionCatalog
for Identifier namespace structure references.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Refactor PR9: Extract decision logic and improve test quality
Changes:
- Extracted shouldUseServerSidePlanning() method with boolean inputs
- Replaced reflection-based test with clean unit test
- Tests all input combinations without brittle reflection code
- Improved testability and maintainability
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Clean up ServerSidePlannedTable: Remove unnecessary helper function
- Remove misleading "Fallback" comment that didn't apply to all cases
- Inline create() function into tryCreate() to reduce indirection
- Simplify logic: directly handle client creation in try-catch
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Remove unnecessary logging and unused imports from ServerSidePlannedTable
- Remove conditional logging that differentiated between forced and fallback paths
- Remove unused imports: MDC and DeltaLogKeys
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Remove useless AutoCloseable implementation from ServerSidePlannedTable
The close() method was never called because:
- Spark's Table interface has no lifecycle hooks
- No code explicitly called close() on ServerSidePlannedTable instances
- No try-with-resources or Using() blocks wrapped it
HTTP connection cleanup happens via connection timeouts (30s) and JVM
finalization, making AutoCloseable purely ceremonial dead code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Simplify verbose test suite comments
Reduced 20+ line formatted comments to simple 2-line descriptions.
The bullet-pointed lists were over-documenting obvious test structure.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Remove useless forTesting() wrapper method
The forTesting() method was just a wrapper around 'new' that added no value.
Tests now directly instantiate ServerSidePlannedTable with the constructor.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Fix brittle test assertion for table capabilities
Removed assertion that table has exactly 1 capability, which would break
if we add streaming support (MICRO_BATCH_READ, CONTINUOUS_READ) or other
non-write capabilities later.
Now tests what actually matters: supports BATCH_READ, does NOT support
BATCH_WRITE.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Remove redundant SupportsWrite interface check in test
Testing !isInstanceOf[SupportsWrite] is redundant with checking
!capabilities.contains(BATCH_WRITE) because:
- BATCH_WRITE capability requires SupportsWrite interface
- Having SupportsWrite without BATCH_WRITE would violate Spark contract
The capability check tests the public API contract, which is sufficient.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Remove e2e/integration terminology from ServerSidePlanningSuite
Changed:
- Test names: removed "E2E:" prefix
- Database name: integration_db → test_db
- Table name: e2e_test → planning_test
These tests use mock clients, not external systems, so e2e/integration
terminology was misleading.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Merge test suites: keep existing ServerSidePlannedTableSuite, delete new ServerSidePlanningSuite
ServerSidePlanningSuite was added in this PR, while ServerSidePlannedTableSuite
existed before. Merged them by keeping the existing file and deleting the new one,
so the PR shows modification rather than deletion+addition.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Refactor tests and improve documentation
- Refactor ServerSidePlannedTableSuite: create database/table once in beforeAll()
- Use minimal early-return pattern in DeltaCatalog with oss-only markers
- Move current snapshot limitation docs to ServerSidePlanningClient interface
- Add UC credential injection link to hasCredentials() documentation
- Lowercase test names and remove redundant client verification
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Simplify current snapshot limitation documentation
Shorten documentation from 4 lines to 1 concise line.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Address PR9 review feedback
Changes:
1. Remove unnecessary afterEach() cleanup - no resource leaks to prevent
2. Make test 2 explicit by setting config=false instead of relying on cleanup
3. Remove redundant test "loadTable() decision logic" - already covered by other tests
4. Add explanation for deltahadoopconfiguration scalastyle suppression
Tests: 4/4 passing, scalastyle clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Address PR9 review comments: improve test quality and cleanup
- Add withServerSidePlanningEnabled helper method to prevent test pollution
- Encapsulates factory and config setup/teardown in one place
- Guarantees cleanup in finally block
- Prevents tests from interfering with each other
- Replace white-box capability check with black-box insert test
- Test actual insert behavior instead of inspecting capabilities
- Verifies inserts succeed without SSP, fail with SSP enabled
- More realistic end-to-end test of read-only behavior
- Remove OSS-only marker comments from DeltaCatalog
- Clean up // oss-only-start and // oss-only-end comments
- Remove unused import (DeltaCatalog)
All tests passing (4/4):
- full query through DeltaCatalog with server-side planning
- verify normal path unchanged when feature disabled
- shouldUseServerSidePlanning() decision logic
- ServerSidePlannedTable is read-only
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
---------
Co-authored-by: Claude <[email protected]>1 parent da0c2ae commit c94cf91
File tree
5 files changed
+331
-104
lines changed- spark/src
- main/scala/org/apache/spark/sql/delta
- catalog
- serverSidePlanning
- sources
- test/scala/org/apache/spark/sql/delta/serverSidePlanning
5 files changed
+331
-104
lines changedLines changed: 10 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| |||
47 | 48 | | |
48 | 49 | | |
49 | 50 | | |
50 | | - | |
| 51 | + | |
51 | 52 | | |
52 | 53 | | |
53 | 54 | | |
| |||
236 | 237 | | |
237 | 238 | | |
238 | 239 | | |
239 | | - | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
240 | 247 | | |
241 | 248 | | |
242 | 249 | | |
| |||
Lines changed: 122 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
35 | 154 | | |
36 | 155 | | |
37 | 156 | | |
| |||
45 | 164 | | |
46 | 165 | | |
47 | 166 | | |
48 | | - | |
| 167 | + | |
| 168 | + | |
49 | 169 | | |
50 | 170 | | |
51 | 171 | | |
| |||
Lines changed: 9 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2905 | 2905 | | |
2906 | 2906 | | |
2907 | 2907 | | |
| 2908 | + | |
| 2909 | + | |
| 2910 | + | |
| 2911 | + | |
| 2912 | + | |
| 2913 | + | |
| 2914 | + | |
| 2915 | + | |
| 2916 | + | |
2908 | 2917 | | |
2909 | 2918 | | |
2910 | 2919 | | |
0 commit comments