[PROTOCOL RFC] New writer feature MaterializePartitionColumns #5555
Description
Protocol Change Request
Description of the protocol change
This protocol change introduces a new writer-only table feature called materializePartitionColumns. When enabled, this feature requires partition columns to be physically materialized in Parquet data files alongside the data columns.
Currently, Delta tables store partition column values primarily in the table metadata (in the partitionValues field of AddFile actions), and by default these columns are not physically written into the Parquet data files themselves. This creates challenges for Parquet readers that access files directly without interpreting Delta metadata, as well as for Iceberg readers which expect partition columns to be present in data files.
This feature will provide a protocol-level mechanism to ensure all writers materialize partition columns during the period when the feature is enabled.
For more details, see the RFC PR.
Willingness to contribute
The Delta Lake Community encourages protocol innovations. Would you or another member of your organization be willing to contribute this feature to the Delta Lake code base?
- Yes. I can contribute.
- Yes. I would be willing to contribute with guidance from the Delta Lake community.
- No. I cannot contribute at this time.