Dataset Config

pydantic model dsgrid.config.dataset_config.DatasetConfigModel

Represents dataset configurations.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• data_classification (dsgrid.config.dataset_config.DataClassificationType)

• data_schema (dsgrid.config.dataset_config.StandardDataSchemaModel | dsgrid.config.dataset_config.OneTableDataSchemaModel)

• data_source (str | None)

• data_source_authors (list[str] | None)

• data_source_date (str | None)

• data_source_doi_url (str | None)

• data_source_version (str | None)

• dataset_id (str)

• dataset_qualifier_metadata (dsgrid.config.dataset_config.QuantityModel | dsgrid.config.dataset_config.GrowthRateModel)

• dataset_type (dsgrid.config.dataset_config.InputDatasetType)

• description (str | None)

• dimension_references (list[dsgrid.config.dimensions.DimensionReferenceModel])

• dimensions (list[dsgrid.config.dimensions.DimensionModel | dsgrid.config.dimensions.DateTimeDimensionModel | dsgrid.config.dimensions.AnnualTimeDimensionModel | dsgrid.config.dimensions.RepresentativePeriodTimeDimensionModel | dsgrid.config.dimensions.IndexTimeDimensionModel | dsgrid.config.dimensions.NoOpTimeDimensionModel])

• enable_unit_conversion (bool)

• origin_contributors (list[str] | None)

• origin_creator (str | None)

• origin_organization (str | None)

• origin_project (str | None)

• sector_description (str | None)

• tags (list[str] | None)

• trivial_dimensions (list[dsgrid.dimension.base_models.DimensionType])

• use_project_geography_time_zone (bool)

• user_defined_metadata (dict[str, Any])

Validators:

• check_dataset_id » dataset_id

• check_files » dimensions

• check_names » dimensions

• check_time_not_trivial » trivial_dimensions

• check_time_zone » all fields

• handle_legacy_fields » all fields

field data_classification: DataClassificationType [Required]

Data security classification (e.g., low, moderate).

Validated by:

• check_time_zone

• handle_legacy_fields

field data_schema: StandardDataSchemaModel | OneTableDataSchemaModel [Required]

Schema (table layouts) used for writing out the dataset

Validated by:

• check_time_zone

• handle_legacy_fields

field data_source: str | None = None

Original data source name, e.g. ‘ComStock’, ‘EIA 861’.

Validated by:

• check_time_zone

• handle_legacy_fields

field data_source_authors: list[str] | None = None

List of authors for the original data source.

Validated by:

• check_time_zone

• handle_legacy_fields

field data_source_date: str | None = None

Date or year the original source data were published, e.g., ‘2021’ for ‘EIA AEO 2021’.

Validated by:

• check_time_zone

• handle_legacy_fields

field data_source_doi_url: str | None = None

Original data source doi or other url

Validated by:

• check_time_zone

• handle_legacy_fields

field data_source_version: str | None = None

Source data version, if applicable. For example, could specify preliminary versus final data.

Validated by:

• check_time_zone

• handle_legacy_fields

field dataset_id: str [Required]

Unique dataset identifier.

Validated by:

• check_dataset_id

• check_time_zone

• handle_legacy_fields

field dataset_qualifier_metadata: QuantityModel | GrowthRateModel = QuantityModel(dataset_qualifier_type=<DatasetQualifierType.QUANTITY: 'quantity'>)

Additional metadata to include related to the dataset_qualifier

Validated by:

• check_time_zone

• handle_legacy_fields

field dataset_type: InputDatasetType = InputDatasetType.UNSPECIFIED

Input dataset type.

Validated by:

• check_time_zone

• handle_legacy_fields

field description: str | None = None

A detailed description of the dataset

Validated by:

• check_time_zone

• handle_legacy_fields

field dimension_references: list[DimensionReferenceModel] = []

List of registered dimension references that make up the dimensions of dataset.

Validated by:

• check_time_zone

• handle_legacy_fields

field dimensions: Annotated[list[DimensionModel | DateTimeDimensionModel | AnnualTimeDimensionModel | RepresentativePeriodTimeDimensionModel | IndexTimeDimensionModel | NoOpTimeDimensionModel], BeforeValidator(func=handle_dimension_union, json_schema_input_type=PydanticUndefined)] = []

List of dimensions that make up the dimensions of dataset. They will be automatically registered during dataset registration and then converted to dimension_references.

Constraints:

• func = <function handle_dimension_union at 0x7f79613d8ae0>

• json_schema_input_type = PydanticUndefined

Validated by:

• check_files

• check_names

• check_time_zone

• handle_legacy_fields

field enable_unit_conversion: bool = True

If the dataset uses its dimension mapping for the metric dimension to also perform unit conversion, then this value should be false.

Validated by:

• check_time_zone

• handle_legacy_fields

field origin_contributors: list[str] | None = None

List of contributors to the compilation of this dataset for dsgrid, e.g., [‘Harry Potter’, ‘Ronald Weasley’]

Validated by:

• check_time_zone

• handle_legacy_fields

field origin_creator: str | None = None

First and last name of the person who formatted this dataset for dsgrid

Validated by:

• check_time_zone

• handle_legacy_fields

field origin_organization: str | None = None

Organization name of the origin_creator, e.g., ‘NREL’

Validated by:

• check_time_zone

• handle_legacy_fields

field origin_project: str | None = None

Name of the project for/from which this dataset was compiled, e.g., ‘IEF’, ‘Building Standard Scenarios’.

Validated by:

• check_time_zone

• handle_legacy_fields

field sector_description: str | None = None

Sectoral description (e.g., residential, commercial, industrial, transportation, electricity)

Validated by:

• check_time_zone

• handle_legacy_fields

field tags: list[str] | None = None

List of data tags

Validated by:

• check_time_zone

• handle_legacy_fields

field trivial_dimensions: list[DimensionType] = []

List of trivial dimensions (i.e., 1-element dimensions) that do not exist in the load_data_lookup. List the dimensions by dimension type. Trivial dimensions are 1-element dimensions that are not present in the parquet data columns. Instead they are added by dsgrid as an alias column.

Validated by:

• check_time_not_trivial

• check_time_zone

• handle_legacy_fields

field use_project_geography_time_zone: bool = False

If true, time zones will be applied from the project’s geography dimension. If false, the dataset’s geography dimension records must provide a time zone column.

Validated by:

• check_time_zone

• handle_legacy_fields

field user_defined_metadata: dict[str, Any] = {}

Additional user defined metadata fields

Validated by:

• check_time_zone

• handle_legacy_fields

validator check_dataset_id  »  dataset_id

Check dataset ID validity

validator check_files  »  dimensions

Validate dimension files are unique across all dimensions

validator check_names  »  dimensions

Validate dimension names are unique across all dimensions.

validator check_time_not_trivial  »  trivial_dimensions

validator check_time_zone  »  all fields

Validate whether required time zone information is present.

validator handle_legacy_fields  »  all fields