Project Config

pydantic model dsgrid.config.project_config.ProjectConfigModel

Represents project configurations

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• datasets (list[dsgrid.config.project_config.InputDatasetModel])

• description (str)

• dimension_mappings (dsgrid.config.project_config.DimensionMappingsModel)

• dimensions (dsgrid.config.project_config.DimensionsModel)

• name (str)

• project_id (str)

• status (dsgrid.registry.common.ProjectRegistryStatus)

Validators:

• check_project_id_handle » project_id

field datasets: list[InputDatasetModel] [Required]

List of input datasets for the project.

field description: str [Required]

Detailed project description.

field dimension_mappings: DimensionMappingsModel = DimensionMappingsModel(base_to_supplemental_references=[], dataset_to_project={})

List of project mappings. Initialized with base-to-base and base-to-supplemental mappings. dataset-to-project mappings are added by dsgrid as datasets get registered with the project.

field dimensions: DimensionsModel [Required]

List of base and supplemental dimensions.

field name: str [Required]

A project name to accompany the ID.

field project_id: str [Required]

A unique project identifier that is project-specific (e.g., ‘standard-scenarios-2021’).

Validated by:

• check_project_id_handle

field status: ProjectRegistryStatus = ProjectRegistryStatus.INITIAL_REGISTRATION

project registry status

validator check_project_id_handle  »  project_id

Check for valid characters in project id

pydantic model dsgrid.config.project_config.InputDatasetModel

Defines an input dataset for the project config.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base_dimension_names (dsgrid.config.project_config.DatasetBaseDimensionNamesModel)

• dataset_id (str)

• dataset_type (dsgrid.config.dataset_config.InputDatasetType)

• mapping_references (list[dsgrid.config.dimension_mapping_base.DimensionMappingReferenceModel])

• required_dimensions (dsgrid.config.project_config.RequiredDimensionsModel)

• status (dsgrid.registry.common.DatasetRegistryStatus)

• time_based_data_adjustment (dsgrid.dimension.time.TimeBasedDataAdjustmentModel)

• version (str)

• wrap_time_allowed (bool)

Validators:

• check_data_adjustment » time_based_data_adjustment

field base_dimension_names: DatasetBaseDimensionNamesModel = DatasetBaseDimensionNamesModel(geography=None, metric=None, model_year=None, scenario=None, sector=None, subsector=None, time=None, weather_year=None)

Defines the project base dimensions to which the dataset will map itself. Auto-populated during submission.

field dataset_id: str [Required]

Unique dataset identifier.

field dataset_type: InputDatasetType [Required]

Dataset type.

field mapping_references: list[DimensionMappingReferenceModel] = []

Defines how to map the dataset dimensions to the project. Auto-populated during submission.

field required_dimensions: RequiredDimensionsModel = RequiredDimensionsModel(single_dimensional=RequiredDimensionRecordsModel(geography=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), metric=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), model_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), scenario=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), sector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), subsector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), weather_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])), multi_dimensional=[])

Defines required record IDs that must exist for each dimension.

field status: DatasetRegistryStatus = DatasetRegistryStatus.UNREGISTERED

Registration status of the dataset, added by dsgrid.

field time_based_data_adjustment: TimeBasedDataAdjustmentModel = TimeBasedDataAdjustmentModel(leap_day_adjustment=<LeapDayAdjustmentType.NONE: 'none'>, daylight_saving_adjustment=DaylightSavingAdjustmentModel(spring_forward_hour=<DaylightSavingSpringForwardType.NONE: 'none'>, fall_back_hour=<DaylightSavingFallBackType.NONE: 'none'>))

Defines how the rest of the dataframe is adjusted with respect to time. E.g., when drop associated data when dropping a leap day timestamp.

Validated by:

• check_data_adjustment

field version: str = None

Version of the registered dataset

field wrap_time_allowed: bool = False

Whether to allow dataset time to be wrapped to project time if different

validator check_data_adjustment  »  time_based_data_adjustment

Check daylight saving adjustment

pydantic model dsgrid.config.project_config.DimensionsModel

Contains dimensions defined by a project

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base_dimension_references (list[dsgrid.config.dimensions.DimensionReferenceModel])

• base_dimensions (list[dsgrid.config.dimensions.DimensionModel | dsgrid.config.dimensions.DateTimeDimensionModel | dsgrid.config.dimensions.AnnualTimeDimensionModel | dsgrid.config.dimensions.RepresentativePeriodTimeDimensionModel | dsgrid.config.dimensions.IndexTimeDimensionModel | dsgrid.config.dimensions.NoOpTimeDimensionModel])

• subset_dimensions (list[dsgrid.config.project_config.SubsetDimensionGroupModel])

• supplemental_dimension_references (list[dsgrid.config.dimensions.DimensionReferenceModel])

• supplemental_dimensions (list[dsgrid.config.supplemental_dimension.SupplementalDimensionModel])

Validators:

• check_dimension_names » all fields

• check_dimensions » all fields

• check_files » base_dimensions

• check_names » base_dimensions

• check_subset_dimensions » subset_dimensions

• check_time_zone » base_dimensions

• pre_check_values » all fields

field base_dimension_references: list[DimensionReferenceModel] = []

List of registry references (DimensionReferenceModel) for a project’s base dimensions.

Validated by:

• check_dimension_names

• check_dimensions

• pre_check_values

field base_dimensions: Annotated[list[DimensionModel | DateTimeDimensionModel | AnnualTimeDimensionModel | RepresentativePeriodTimeDimensionModel | IndexTimeDimensionModel | NoOpTimeDimensionModel], BeforeValidator(func=handle_dimension_union, json_schema_input_type=PydanticUndefined)] = []

List of dimensions for a project’s base dimensions. They will be automatically registered during project registration and then converted to base_dimension_references.

Constraints:

• func = <function handle_dimension_union at 0x7fe7085e7ce0>

• json_schema_input_type = PydanticUndefined

Validated by:

• check_dimension_names

• check_dimensions

• check_files

• check_names

• check_time_zone

• pre_check_values

field subset_dimensions: list[SubsetDimensionGroupModel] = []

List of subset dimension groups

Validated by:

• check_dimension_names

• check_dimensions

• check_subset_dimensions

• pre_check_values

field supplemental_dimension_references: list[DimensionReferenceModel] = []

List of registry references for a project’s supplemental dimensions.

Validated by:

• check_dimension_names

• check_dimensions

• pre_check_values

field supplemental_dimensions: list[SupplementalDimensionModel] = []

List of supplemental dimensions. They will be automatically registered during project registration and then converted to supplemental_dimension_references.

Validated by:

• check_dimension_names

• check_dimensions

• pre_check_values

validator check_dimension_names  »  all fields

Check that all dimension query names are unique.

validator check_dimensions  »  all fields

Validate that the dimensions are complete and consistent.

validator check_files  »  base_dimensions

Validate dimension files are unique across all dimensions

validator check_names  »  base_dimensions

Validate dimension names are unique across all dimensions.

validator check_subset_dimensions  »  subset_dimensions

Check that each subset dimension has a unique name.

validator check_time_zone  »  base_dimensions

Validate the time zone column in geography records.

validator pre_check_values  »  all fields

Checks that base dimensions are defined.

pydantic model dsgrid.config.project_config.DimensionReferenceModel

Reference to a dimension stored in the registry

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• dimension_id (str)

• dimension_type (dsgrid.dimension.base_models.DimensionType)

• version (str)

field dimension_id: str [Required]

Unique ID of the dimension in the registry

field dimension_type: DimensionType [Required] (alias 'type')

Type of the dimension

field version: str [Required]

Version of the dimension

pydantic model dsgrid.config.project_config.RequiredDimensionsModel

Defines required record IDs that must exist for each dimension in a dataset. Record IDs can reside in the project’s base or subset dimensions.

Requirements can be specified for a single dimension or a combination of dimensions. For example, if a project includes commercial, residential, and transportation sectors but the dataset has only transporation sector records, it should specify a single_dimensional requirement that is a subset of of the project’s base dimension. {“single_dimensional”: “sector”: {“base”: {“record_ids”: [“transportation”]}}}.

If a dataset’s requirements span multiple dimensions, such as if it does not have some metric records for some geography records, then a multi_dimensional requirement should be specified. (By default, a full cross join is assumed to be present.) `{“multi_dimensional”: {

“geography”: {“base”: {“record_ids”: [“12345”]}} “metric”: {“base”: {“record_ids”: [“electricity_cooling”]}}

}

}`

If a dataset specifies a dimension type within a multi_dimensional section and wants to use all records from a project base dimension, it can specify `base.record_ids = [“__all__”] as a shorthand notation.

Requirements for a dimension cannot be defined in both single_dimensional and multi_dimensional sections.

If no records are listed for a dimension then all project base records are required.

It might be easier for a dataset to specify what it does not have rather than what it does have. In that case, it is recommended to use the RequiredDimensionRecordsModel.base_missing field. dsgrid will compute the difference of the base dimension records and the base_missing records to determine the dataset’s required records.

If a project has multiple base dimensions of the same type, the RequiredDimensionRecordsModel.dimension_name must be specified to identify the base dimension that contains the record IDs.

If a dataset contains a subset of project base dimension records that are defined in the project’s subset dimensions, it is recommended to use that specification. dsgrid will substitute base records for mapped subset records at runtime.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• multi_dimensional (list[dsgrid.config.project_config.RequiredDimensionRecordsModel])

• single_dimensional (dsgrid.config.project_config.RequiredDimensionRecordsModel)

Validators:

• check_for_duplicates » all fields

field multi_dimensional: list[RequiredDimensionRecordsModel] = []

Required records for a combination of dimensions. For example, there may be a dataset requirement for only one subsector for a given sector instead of a cross product.

Validated by:

• check_for_duplicates

field single_dimensional: RequiredDimensionRecordsModel = RequiredDimensionRecordsModel(geography=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), metric=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), model_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), scenario=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), sector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), subsector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), weather_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]))

Required records for a single dimension.

Validated by:

• check_for_duplicates

validator check_for_duplicates  »  all fields

1. Ensure that the same dimension does not have requirements in both single and multi dimensional sections.

2. Set any dimensions that do not have specifications to require all base dimension records (as long as there is only one project base dimension).

pydantic model dsgrid.config.project_config.DimensionMappingsModel

Defines all dimension mappings associated with a dsgrid project, including base-to-supplemental mappings and dataset-to-project mappings.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base_to_supplemental_references (list[dsgrid.config.dimension_mapping_base.DimensionMappingReferenceModel])

• dataset_to_project (dict[str, list[dsgrid.config.dimension_mapping_base.DimensionMappingReferenceModel]])

field base_to_supplemental_references: list[DimensionMappingReferenceModel] = []

Base dimension to supplemental dimension mappings (e.g., county-to-state) used to support various queries and dimension transformations.

field dataset_to_project: dict[str, list[DimensionMappingReferenceModel]] = {}

Dataset-to-project mappings map dataset dimensions to project dimensions.

pydantic model dsgrid.config.project_config.RequiredDimensionRecordsModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• geography (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• metric (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• model_year (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• scenario (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• sector (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• subsector (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• weather_year (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

field geography: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])

field metric: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])

field model_year: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])

field scenario: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])

field sector: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])

field subsector: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])

field weather_year: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])

pydantic model dsgrid.config.project_config.RequiredSupplementalDimensionRecordsModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• name (str)

• record_ids (list[str])

field name: str [Required]

Name of a supplemental dimension

field record_ids: list[str] [Required]

One or more record IDs in the supplemental dimension

pydantic model dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base (dsgrid.config.project_config.RequiredBaseDimensionModel)

• base_missing (dsgrid.config.project_config.RequiredBaseDimensionModel)

• subset (list[dsgrid.config.project_config.RequiredSubsetDimensionRecordsModel])

Validators:

• check_base » all fields

• handle_legacy_format » all fields

field base: RequiredBaseDimensionModel = RequiredBaseDimensionModel(record_ids=[], dimension_name=None)

Validated by:

• check_base

• handle_legacy_format

field base_missing: RequiredBaseDimensionModel = RequiredBaseDimensionModel(record_ids=[], dimension_name=None)

Validated by:

• check_base

• handle_legacy_format

field subset: list[RequiredSubsetDimensionRecordsModel] = []

Validated by:

• check_base

• handle_legacy_format

validator check_base  »  all fields

defines_dimension_requirement() → bool

Returns True if the model defines a dimension requirement.

validator handle_legacy_format  »  all fields

pydantic model dsgrid.config.project_config.SubsetDimensionGroupModel

Defines one or more subset dimension selectors for a dimension type.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base_dimension_name (str | None)

• create_supplemental_dimension (bool)

• description (str)

• dimension_type (dsgrid.dimension.base_models.DimensionType)

• filename (str | None)

• name (str)

• record_ids (set[str])

• selector_references (list[dsgrid.config.dimensions.DimensionReferenceModel])

• selectors (list[dsgrid.config.project_config.SubsetDimensionSelectorModel])

Validators:

• check_selectors » selectors

• load_records » all fields

field base_dimension_name: str | None = None

Name of base dimension for the supplemental dimension mapping, if create_supplemental_dimension is true. Required if there are multiple base dimensions for this type.

Validated by:

• load_records

field create_supplemental_dimension: bool = True

Auto-generate supplemental dimensions in order to allow aggregrations on the subsets.

Validated by:

• load_records

field description: str [Required]

Validated by:

• load_records

field dimension_type: DimensionType [Required] (alias 'type')

Type of the dimension

Validated by:

• load_records

field filename: str | None = None (alias 'file')

Filename containing dimension records. Only populated for initial registration. Each selector’s records are stored as JSON objects in the dsgrid registry.

Validated by:

• load_records

field name: str [Required]

Validated by:

• load_records

field record_ids: set[str] = {}

Validated by:

• load_records

field selector_references: list[DimensionReferenceModel] = []

References to the subset dimensions generated by dsgrid during registration.

Validated by:

• load_records

field selectors: list[SubsetDimensionSelectorModel] [Required]

Dimension selectors

Validated by:

• check_selectors

• load_records

validator check_selectors  »  selectors

Check that the selectors are defined consistently.

validator load_records  »  all fields

Load the records for each subset dimension selector.

pydantic model dsgrid.config.project_config.SubsetDimensionGroupListModel

Defines a list of subset dimensions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• subset_dimensions (List[dsgrid.config.project_config.SubsetDimensionGroupModel])

field subset_dimensions: Annotated[List[SubsetDimensionGroupModel], Len(min_length=1, max_length=None)] [Required]

List of subset dimensions to be registered

Constraints:

• min_length = 1

To add new supplemental dimensions

pydantic model dsgrid.config.supplemental_dimension.SupplementalDimensionsListModel

Defines a list of supplemental dimensions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• supplemental_dimensions (List[dsgrid.config.supplemental_dimension.SupplementalDimensionModel])

field supplemental_dimensions: Annotated[List[SupplementalDimensionModel], Len(min_length=1, max_length=None)] [Required]

List of supplemental dimensions and mappings to be registered

Constraints:

• min_length = 1

To add new dataset requirements

pydantic model dsgrid.config.input_dataset_requirements.InputDatasetListModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• datasets (List[dsgrid.config.project_config.InputDatasetModel])

field datasets: Annotated[List[InputDatasetModel], Len(min_length=1, max_length=None)] [Required]

List of input datasets for the project.

Constraints:

• min_length = 1

To replace dataset dimension requirements

pydantic model dsgrid.config.input_dataset_requirements.InputDatasetDimensionRequirementsListModel

Defines a list of dataset dimension requirements.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• dataset_dimension_requirements (List[dsgrid.config.input_dataset_requirements.InputDatasetDimensionRequirementsModel])

field dataset_dimension_requirements: Annotated[List[InputDatasetDimensionRequirementsModel], Len(min_length=1, max_length=None)] [Required]

List of dataset dimension requirements

Constraints:

• min_length = 1

pydantic model dsgrid.config.input_dataset_requirements.InputDatasetDimensionRequirementsModel

Defines dataset dimension requirements.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• dataset_id (str)

• required_dimensions (dsgrid.config.project_config.RequiredDimensionsModel)

field dataset_id: str [Required]

field required_dimensions: RequiredDimensionsModel [Required]

Defines required record IDs that must exist for each dimension.