to_msgpack() error: E TypeError: can not serialize 'ExternalPartition' object

[gshank]

• mashumaro version: 3.15
• Python version: 3.11.8
• Operating System: MacOS

Description

A to_msgpack() call that worked in 3.14 no longer works for a particular class in 3.15.

What I Did

test script:

from dataclasses import dataclass, field
from typing import Dict, Any, Optional, Union, List

from mashumaro.mixins.msgpack import DataClassMessagePackMixin

@dataclass
class ExternalPartition(DataClassMessagePackMixin):
    name: str = ""
    description: str = ""
    data_type: str = ""
    meta: Dict[str, Any] = field(default_factory=dict)

@dataclass 
class ExternalTable(DataClassMessagePackMixin):
    location: Optional[str] = None
    file_format: Optional[str] = None
    row_format: Optional[str] = None
    tbl_properties: Optional[str] = None
    partitions: Optional[Union[List[ExternalPartition], List[str]]] = None

def test_partitions_serialization():

    part1 = ExternalPartition(
        name="partition 1",
        description="partition 1",
        data_type="string",
    )

    part2 = ExternalPartition(
        name="partition 2",
        description="partition 2",
        data_type="string",
    )

    ext_table = ExternalTable(
        location="my_location",
        file_format="my file format",
        row_format="row format",
        partitions=[part1, part2],
    )

    ext_table_dict = ext_table.to_dict()
    assert isinstance(ext_table_dict["partitions"][0], dict)

    ext_table_msgpack = ext_table.to_msgpack()
    assert ext_table_msgpack

output

>       ext_table_msgpack = ext_table.to_msgpack()

tests/functional/artifacts/test_serialization.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
<string>:12: in __mashumaro_to_msgpack__
    ???
../mashumaro/mashumaro/mixins/msgpack.py:30: in default_encoder
    return msgpack.packb(data, use_bin_type=True)
env/lib/python3.11/site-packages/msgpack/__init__.py:36: in packb
    return Packer(**kwargs).pack(o)
msgpack/_packer.pyx:279: in msgpack._cmsgpack.Packer.pack
    ???
msgpack/_packer.pyx:276: in msgpack._cmsgpack.Packer.pack
    ???
msgpack/_packer.pyx:270: in msgpack._cmsgpack.Packer._pack
    ???
msgpack/_packer.pyx:213: in msgpack._cmsgpack.Packer._pack_inner
    ???
msgpack/_packer.pyx:270: in msgpack._cmsgpack.Packer._pack
    ???
msgpack/_packer.pyx:232: in msgpack._cmsgpack.Packer._pack_inner
    ???
msgpack/_packer.pyx:270: in msgpack._cmsgpack.Packer._pack
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: can not serialize 'ExternalPartition' object

msgpack/_packer.pyx:257: TypeError

msgpack is 1.1.0

[gshank]

Actually, it looks like this fails with 3.14 also. In 3.14 with the order Union[List[str], List[ExternalPartition]] the List[ExternalPartition] doesn't always serialize properly (it stays an object). In other cases it's okay (not sure what the difference is). In 3.15 that Union order causes the output to be a stringified object instead of a dictionary, so when I changed it to Union[List[ExternalPartition], List[str]] it converted to a dict okay, but we got the msgpack failure.

[Fatal1ty]

This is certainly a bug related to no_copy_collections parameter. The condition for no copying should be improved because it's now very naive at the moment:

mashumaro/mashumaro/core/meta/types/pack.py

Lines 804 to 805 in 19d4642

	elif spec.origin_type is list:
	return f"{spec.expression}.copy()"

Current workaround is to use your own dialect with the empty no_copy_collections tuple (or just make it so right in MessagePackDialect before the first import of your dataclasses 👀):

from mashumaro.mixins.msgpack import (
    DataClassMessagePackMixin,
    MessagePackDialect,
)

class FixedMessagePackDialect(MessagePackDialect):
    no_copy_collections = ()

@dataclass
class ExternalTable(DataClassMessagePackMixin):
    location: Optional[str] = None
    file_format: Optional[str] = None
    row_format: Optional[str] = None
    tbl_properties: Optional[str] = None
    partitions: Optional[Union[List[ExternalPartition], List[str]]] = None

    class Config(BaseConfig):
        dialect = FixedMessagePackDialect

This fixed dialect can be used in other ways, for example, as the to_msgpack dialect argument with the dialect keyword argument enabled.