RFC to introduce an IO Client abstraction layer

[desmondcheongzx]

Background

Daft currently has several custom IO connectors (S3, GCS, etc.), each integrated directly into the main codebase. This creates friction for contributors who wish to add new storage backends (like ByteDance’s TOS) or even proprietary IO clients, and increases the maintenance burden for the core team.

To improve modularity and sustainability, we propose a unified IO abstraction layer based on fsspec or OpenDAL, which allows third-party connectors to live and evolve independently in an external library outside of Daft core.

Goals

1. Decouple core Daft from storage backends
   Core Daft should depend only on an abstract file system interface, and potentially a select few IO clients like S3 that we can test and maintain easily.

2. Empower external contributors
   Developers can build and publish new connectors like daft-tos or daft-hdfs as standalone packages.

3. Leverage community ecosystems
   Use existing fsspec/OpenDAL backends where possible, and provide a thin Daft-specific adapter.

4. Simplify long-term maintenance
   New storage systems can be added without changing Daft’s main codebase. Key stakeholders can also own and iterate on their connectors with full discretion.

Proposed Architecture

Option A: fsspec-based

• Adopt fsspec.AbstractFileSystem as the interface for file IO.
• Each backend implements a subclass (e.g. TOSFileSystem).
• Daft’s IO code operates on generic fsspec URLs (tos://bucket/path).
• Backends can be distributed separately as daft-tos, daft-minio, etc.

Option B: OpenDAL-based

• Integrate OpenDAL via its Python bindings.
• OpenDAL already supports many cloud/object stores (including S3-compatible systems).
• Daft can define an OpenDALIOClient that wraps OpenDAL’s API and exposes Daft’s dataframes to any supported backend.

Migration Plan

Phase 1: Maintain existing built-in clients for S3, GCS, etc.

Phase 2: Introduce experimental fsspec/OpenDAL adapter and migrate TOS client onto it.

Phase 3: Deprecate built-in clients once equivalent external connectors exist.

Phase 4: Document the plugin interface and create a registry for community-maintained connectors.

Next Steps

• Align on fsspec vs. OpenDAL as the preferred base layer.

• Define a minimal interface (open/read/write/list) Daft expects.

• Collaborate with ByteDance to pilot the first external connector (daft-tos).

• Publish a design doc + example repo to guide future contributors.

[stayrascal]

@desmondcheongzx Nice proposal. It's convenient to developers to extend more storage backend.

There are some questions about the proposed architectures mentioned above all:

• Regarding fsspec, as I know it might popular used in Python stack, if we adapt this option, will we create a python fs interface and implement the fs via rust and bind them via pyO3 ? And also I think Daft already used some fsspec implementation, e.g. PyarrowFileSystem, S3FileSystem, which also brings a problem that some IO behaviors are through Daft native IO, some IO behaviors are through fsspec, we should unify it.
• Regarding OpenDAL, it's a nice project and has broader ecosystem, but it's a little bit heavy for Daft from my perspective, first, Daft's IO pattern is based on object semantic instead of filesystem semantic, second Daft only depends on a few APIs, e.g. putObject, getObject, headObject, MPU, they are a sub APIs of OpenDAL.

Take a summary, currently Daft might need a object interface not a fs interface, actually I think the current design of ObjectSource trait is concise(might need improve the docs to describe its semantic), we can reuse the trait, and decouple the each storage backend to separate crates. But the IOConfig need to be refactored, I think it's a main pain point during add a new storage backend, because it combined all storage backend configuration, if we want to add a new storage backend, we have to modify this class, or if we want to add a common property, e.g. support suffix get, we have to add it in each sub xxConfig or add a new API on ObjectSource, it might not a good design. If we are trying to refactor the IO layer, IOConfig might be the first step.