Backward Compatibility Policy

[westonpace]

Since I started #5099 I figured I'd start another discussion for the other half of the issue.

A backwards compatibility failure is when data written by an older version is no longer readable by a newer version. Backwards compatibility is extremely important for databases. Once data is written users expect to be able to read it continuously. Requiring migration of data is often extremely costly and expensive.

So far, Lance has tried very hard (and at the expensive of some amount of technical debt) to ensure we never have backwards compatibility failures.

Question

What should our policy be here? Should we require a data migration of some kind if a dataset is more than X versions / months old?

Proposal

• Lance is always backwards compatible
• Lance can break backwards compatibility on major releases if deemed significant enough

[westonpace]

Personally, my two cents (in the current moment at least), is that we should always be backwards compatible. I think the bar for breaking backwards compatibility needs to be very very high. I say this as someone who has been maintaining a significant amount of technical debt in the scanner for reading 0.1 files which haven't been the default file version in over a year.

However, I wouldn''t completely rule it out, but a break would probably need to require at least 1 year (if not more) of advance notice.

[jackye1995]

+1. I think if we really want to break backwards compatibility at format level we will have to do a new format major version like what Iceberg does, and even with that the migration period is still long and many aspects are continuing to be backwards compatible and definitely more than 1 year.

[HaochengLIU]

Piggyback on this issue. I agree that we should always be backwards compatible, meanwhile what's the stabilization policy for the new features added to Lance? I'm wondering if there is X month/ Y version(s) that allows the new features to mature.
e.g. recently I realize some data types I added to zonemap/bloomfilter are not accurate. I want to fix them now before it's too late, but donno the paradigm here.

[jackye1995]

I think in general we should add some feature flag like "experimental" for such features so people are aware this is not fully battle tested yet and could be buggy in production use cases.

Another aspect is that these are technically format changes to add a new index type (until we make index pluggable), and we should have flushed out the aspects like type system integration during spec design phase, not just jump to implementation. But at the time of the initial contribution, we did not really have the right process and content (the format doc was very out of date) to really do that. Hopefully now things are caught up we should start to do that for newer format change proposals.