[Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 #5616

allisonport-db · 2025-12-02T21:53:34Z

Which Delta project/connector is this regarding?

Description

PART OF #5326

Contains the following changes:

Removes Spark 3.5 support
Adds explicit Spark 4.0 support
Removes a "master" build for now
Merges shims from the 3.5 vs 4.0 breaking changes into the src code

In a future PR

we will add Spark 4.1.0-SNAPSHOT support (in preparation for the Spark 4.1 release)
we will add back a "master" build tracking Spark master
(these will require adding new shims, but in different areas)

How was this patch tested?

Unit tests + ran integration tests locally (python, scala + pip)

Tracking open TODOs at #5326

This reverts commit 8fd880f.

allisonport-db · 2025-12-05T23:07:21Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/checkpoints/Checkpointer.java

      numberOfAddFiles = checkpointDataIter.getNumberOfAddActions();
    } catch (FileAlreadyExistsException faee) {
      throw new CheckpointAlreadyExistsException(version);
+    } catch (IOException io) {


Upgrading the hadoop version changes this error class

Hm .. I wonder what the change was?

I'm not sure the change in hadoop but instead of seeing a FileAlreadyExistsException we see a IOException with cause FileAlreadyExistsException. We have this tested (at least one test fails w/out this fix here)

They seem similar enough so didn't look further into it. Seems like a minor API difference.

allisonport-db · 2025-12-05T23:08:41Z

In theory I could do the src code shims + test code shims separately if that would help. Let me know if that makes reviews easier (not sure if anyone wants to review the shim code changes closely, or if tests pass & code compiles that's enough).

kernel-spark/src/main/java/io/delta/kernel/spark/catalog/SparkTable.java

scottsand-db · 2025-12-06T00:43:27Z

.github/workflows/kernel_test.yaml

            echo "❌ Cache MISS - will download dependencies"
          fi
-      - name: Run tests
+      # Run unit tests with JDK 17. These unit tests depend on Spark, and Spark 4.0+ is JDK 17.


would this comment be better placed besides java-version: "17" ?

scottsand-db · 2025-12-06T00:45:47Z

.github/workflows/spark_master_test.yaml

@@ -1,59 +0,0 @@
-name: "Delta Spark Master"


I suggest we update the PR title to say drop support for spark 3.5 and spark master compilation ?

I mean we haven't actually been compiling with spark master in a while... (as we're using a very stale snapshot). But I can make the title more clear

Hm. Sorry, I'm still confused. Here we are deleting our job to compile against spark "master" right? (perhaps it was a stale master ..)

But does Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 reflect that?

That seems like an important highlight, sorry, and I want to make sure my understanding is correct

I think calling it spark master before was misleading, in fact, in the previous PR we renamed the spark version spec to spark40Snapshot instead of master. I think saying we are removing spark master is misleading considering we never were compiling with Spark master. We will be fixing that in future PRs.

It would be more correct to say spark_master_test.yaml was incorrectly named this whole time.

setup.py

allisonport-db · 2025-12-08T23:17:42Z

project/SparkMimaExcludes.scala

+
+      // Changes in 4.1.0
+      // TODO: change in type hierarchy due to removal of DeltaThrowableConditionShim
+      ProblemFilters.exclude[MissingTypesProblem]("io.delta.exceptions.*")


@reviewers this seems safe to me, considering no one should be catching DeltaThrowableConditionShim... but would like additional opinions

allisonport-db · 2025-12-08T23:21:03Z

build.sbt

  ).configureUnidoc()

+/*
+TODO: readd delta-iceberg on Spark 4.0+


@lzlfred Hey Fred, we will be releasing in on both Spark 4.0 and Spark 4.1 next release, we will need to update this build to work for that

also tracking the todo at #5326

allisonport-db · 2025-12-09T00:48:00Z

build.sbt

  ).configureUnidoc()

+/*
+TODO: compilation broken for Spark 4.0


tracking at #5326

@linzhou-db @littlegrasscao FYI can you please look into fixing this once I merge this PR

allisonport-db · 2025-12-09T00:48:47Z

examples/scala/build.sbt

 val lookupSparkVersion: PartialFunction[(Int, Int), String] = {
-  // version 4.0.0-preview1
-  case (major, minor) if major >= 4 => "4.0.0-preview1"
+  // TODO: how to run integration tests for multiple Spark versions


tracking at #5326

allisonport-db · 2025-12-09T00:49:03Z

setup.py

 with open("python/README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

+# TODO: once we support multiple Spark versions update this to be compatible with both


tracking at #5326

allisonport-db added 16 commits December 2, 2025 13:48

Initial build changes

3501613

remove 3.5 shims

77d305b

Remove LoggingShims and DeltaSqlParserShims

83ebe6b

Variant shim

90e1e1f

DecimalPrecisionTypeCoercionShims

2a0f940

LogKeyShims

86fea31

Some more shims

20f72f3

TypeWideningShim

77a9a42

MergeIntoMaterializeSourceShims

13ac618

Throwable, TimeTravelSpec, LogicalRelation

5abacfa

DeltaInvariantCheckerExecShims

a8c2804

Misc

e3dd2b9

Misc shims

ec34fd8

Remove connectors

8fd880f

Update test_cross_spark_publish.py

70effca

Revert "Remove connectors"

11cec06

This reverts commit 8fd880f.

allisonport-db changed the title ~~[Spark][Infra][WIP] Drop support for Spark 3.5 in master~~ [Spark][Infra] Drop support for Spark 3.5 in master Dec 3, 2025

allisonport-db added 4 commits December 2, 2025 18:03

Remove shim dir from build

73deabf

add default

c9dc7fa

drop iceberg for now

36b8472

Misc job fixes

9e41fec

allisonport-db requested review from raveeram-db, scottsand-db and tdas December 3, 2025 02:40

allisonport-db added 6 commits December 2, 2025 19:33

Fix kernel tests

ee907b7

Fix mima for now

73bea33

Type widening shims

6aa8f58

SnapshotManagementSuiteShims

1c58630

DeltaGenerateSymlinkManifestSuiteShims

6bb2c54

DeltaHistoryManagerSuiteShims

d1f8616

allisonport-db added 8 commits December 4, 2025 17:26

MergeIntoMaterializeSourceShims

da50359

ImplicitDMLCastingSuiteShims

f8b0508

MergeIntoMetricsShims

a0a4d27

Structured logging tests

5ae7ee7

Variant suites

219c0a5

Version test shims

17319b5

Final shim fixes

3e3317b

log4j configs

603ab3f

allisonport-db commented Dec 5, 2025

View reviewed changes

raveeram-db requested a review from TimothyW553 December 5, 2025 23:12

Merge remote-tracking branch 'delta-io/master' into update-build-1

4bc7d73

scottsand-db reviewed Dec 6, 2025

View reviewed changes

kernel-spark/src/main/java/io/delta/kernel/spark/catalog/SparkTable.java Outdated Show resolved Hide resolved

scottsand-db reviewed Dec 6, 2025

View reviewed changes

Fix scalastyle

a6fd612

allisonport-db changed the title ~~[Spark][Infra] Drop support for Spark 3.5 in master~~ [Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 Dec 6, 2025

scottsand-db reviewed Dec 6, 2025

View reviewed changes

setup.py Outdated Show resolved Hide resolved

allisonport-db added 2 commits December 8, 2025 13:16

Fix sharing test compile

0de65e8

Merge remote-tracking branch 'delta-io/master' into update-build-1

8d6d117

allisonport-db commented Dec 8, 2025

View reviewed changes

allisonport-db added 2 commits December 8, 2025 15:27

Update setup.py

4dbad62

Skip iceberg test for now

f8c5c3f

allisonport-db mentioned this pull request Dec 9, 2025

[Infra][Feature Request] Upgrade master branch to support Spark 4.0+ and remove remaining standalone-based connectors #5326

Open

5 tasks

Fixes

264e629

allisonport-db requested a review from scottsand-db December 9, 2025 00:46

allisonport-db commented Dec 9, 2025

View reviewed changes

[Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 #5616

Are you sure you want to change the base?

[Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 #5616

Conversation

allisonport-db commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

allisonport-db commented Dec 5, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

allisonport-db Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

allisonport-db commented Dec 2, 2025 •

edited

Loading

allisonport-db Dec 9, 2025 •

edited

Loading