Dataset Mapping Plan

pydantic model dsgrid.query.dataset_mapping_plan.DatasetMappingPlan

Defines how to map a dataset to a list of dimensions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• apply_fraction_op (dsgrid.query.dataset_mapping_plan.MapOperation)

• apply_scaling_factor_op (dsgrid.query.dataset_mapping_plan.MapOperation)

• convert_units_op (dsgrid.query.dataset_mapping_plan.MapOperation)

• dataset_id (str)

• keep_intermediate_files (bool)

• map_time_op (dsgrid.query.dataset_mapping_plan.MapOperation)

• mappings (list[dsgrid.query.dataset_mapping_plan.MapOperation])

Validators:

• check_names » all fields

field apply_fraction_op: MapOperation = MapOperation(name='apply_fraction_op', handle_data_skew=False, persist=False, mapping_reference=None)

Defines handling of the query that applies the from_fraction value after mapping all dimensions.

Validated by:

• check_names

field apply_scaling_factor_op: MapOperation = MapOperation(name='apply_scaling_factor_op', handle_data_skew=False, persist=False, mapping_reference=None)

Defines handling of the query that applies the scaling factor, if one exists. This happens after apply_fraction_op.

Validated by:

• check_names

field convert_units_op: MapOperation = MapOperation(name='convert_units_op', handle_data_skew=False, persist=False, mapping_reference=None)

Defines handling of the query that converts units. This happens after apply_fraction_op and before mapping time. It is strongly recommended to not persist this table because the code currently always persists before mapping time.

Validated by:

• check_names

field dataset_id: str [Required]

ID of the dataset to be mapped.

Validated by:

• check_names

field keep_intermediate_files: bool = False

If True, keep the intermediate tables created during the mapping process. This is useful for debugging and benchmarking, but will consume more disk space.

Validated by:

• check_names

field map_time_op: MapOperation = MapOperation(name='map_time', handle_data_skew=False, persist=False, mapping_reference=None)

Defines handling of the query that maps the time dimension. This happens after convert_units_op. Unlike the other dimension mappings, this does not use the generic mapping code. It relies on specific handling in chronify by time type.

Validated by:

• check_names

field mappings: list[MapOperation] = []

Defines how to map each dimension of the dataset.

Validated by:

• check_names

validator check_names  »  all fields

compute_hash() → str

Compute a hash of the mapping plan.

list_mapping_operations() → list[MapOperation]

List all mapping operations in the plan, in order.

pydantic model dsgrid.query.dataset_mapping_plan.MapOperation

Defines one mapping operation for a dataset.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• handle_data_skew (bool | None)

• mapping_reference (dsgrid.config.dimension_mapping_base.DimensionMappingReferenceModel | None)

• name (str)

• persist (bool)

field handle_data_skew: bool | None = None

Use a salting technique to handle data skew in this mapping operation. Skew can happen when some partitions have significantly more data than others, resulting in unbalanced task execution times. If this value is None, dsgrid will make its own determination of whether this should be done based on the characteristics of the mapping operation. Setting it to True or False will override that behavior and inform dsgrid of what to do. This will automatically trigger a persist to the filesystem (implicitly setting persist to True).

field mapping_reference: DimensionMappingReferenceModel | None = None

Reference to the model used to map the dimension. Set at runtime by dsgrid.

field name: str [Required]

Identifier for the mapping operation. This must be a unique name.

field persist: bool = False

Persist the intermediate dataset to the filesystem after mapping this dimension. This can be useful to prevent the query from becoming too large. It can also be useful for benchmarking and debugging purposes.

pydantic model dsgrid.query.dataset_mapping_plan.MapOperationCheckpoint

Defines a completed mapping operation that has been persisted to the filesystem.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• completed_operation_names (list[str])

• dataset_id (str)

• mapping_plan_hash (str)

• persisted_table_filename (pathlib.Path)

• timestamp (datetime.datetime)

field completed_operation_names: list[str] [Required]

Names of the completed mapping operations.

field dataset_id: str [Required]

field mapping_plan_hash: str [Required]

Hash of the mapping plan. This is used to ensure that the mapping plan hasn’t changed.

field persisted_table_filename: Path [Required]

Path to a persisted file.

field timestamp: datetime [Optional]

Timestamp of when the operation was completed.