Skip to content

Data Types

Data types supported by DataChain must be of type DataType. DataType includes most Python types supported in Pydantic fields, as well as any class that inherits from Pydantic BaseModel.

Pydantic models can be used to group and nest multiple fields together into a single type object. Any Pydantic model must be registered so that the chain knows the expected schema of the model. Alternatively, models may inherit from DataModel, which is a lightweight wrapper around Pydantic's BaseModel that automatically handles registering the model.

DataModel

Bases: BaseModel

Pydantic model wrapper that registers model with DataChain.

__pydantic_init_subclass__ classmethod 

__pydantic_init_subclass__()

It automatically registers every declared DataModel child class.

Source code in datachain/lib/data_model.py 
35
36
37
38
@classmethod
def __pydantic_init_subclass__(cls):
    """It automatically registers every declared DataModel child class."""
    ModelStore.register(cls)

hidden_fields classmethod 

hidden_fields() -> list[str]

Returns a list of fields that should be hidden from the user.

Source code in datachain/lib/data_model.py 
49
50
51
52
@classmethod
def hidden_fields(cls) -> list[str]:
    """Returns a list of fields that should be hidden from the user."""
    return cls._hidden_fields

register staticmethod 

register(models: DataType | Sequence[DataType])

For registering classes manually. It accepts a single class or a sequence of classes.

Source code in datachain/lib/data_model.py 
40
41
42
43
44
45
46
47
@staticmethod
def register(models: DataType | Sequence[DataType]):
    """For registering classes manually. It accepts a single class or a sequence of
    classes."""
    if not isinstance(models, Sequence):
        models = [models]
    for val in models:
        ModelStore.register(val)

DataType module-attribute 

DataType = type[BaseModel] | StandardType

is_chain_type 

is_chain_type(t: type) -> bool

Return true if type is supported by DataChain.

Source code in datachain/lib/data_model.py 
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
def is_chain_type(t: type) -> bool:
    """Return true if type is supported by `DataChain`."""
    if ModelStore.is_pydantic(t):
        return True
    if any(t is ft or t is get_args(ft)[0] for ft in get_args(StandardType)):
        return True

    orig = get_origin(t)
    args = get_args(t)
    if orig is list and len(args) == 1:
        return is_chain_type(get_args(t)[0])

    if orig in (Union, types.UnionType) and len(args) == 2 and (type(None) in args):
        return is_chain_type(args[0] if args[1] is type(None) else args[1])

    return False