Project Config¶

ProjectConfigModel¶

dsgrid.config.project_config.ProjectConfigModel

Represents project configurations

Fields¶

Name	Type	Default	Description
`id`	`int` \| None	`None`	Registry database ID
`version`	`str` \| None	`None`	Version, generated by dsgrid
`project_id`	`str`	(required)	A unique project identifier that is project-specific (e.g., ‘standard-scenarios-2021’).
`name`	`str`	(required)	A project name to accompany the ID.
`description`	`str`	(required)	Detailed project description.
`status`	ProjectRegistryStatus	`"Initial Registration"`	project registry status
`datasets`	list[InputDatasetModel]	(required)	List of input datasets for the project.
`dimensions`	DimensionsModel	(required)	List of `base` and `supplemental` dimensions.
`dimension_mappings`	DimensionMappingsModel	`base_to_supplemental_references=[] dataset_to_project={}`	List of project mappings. Initialized with base-to-base and base-to-supplemental mappings. dataset-to-project mappings are added by dsgrid as datasets get registered with the project.

Validators¶

Name	Applies To	Description
`check_project_id_handle`	`check_project_id_handle`	Check for valid characters in project id

DatasetBaseDimensionNamesModel¶

dsgrid.config.project_config.DatasetBaseDimensionNamesModel

Defines the query names for project base dimensions to which datasets will be mapped. This is important for cases where a project has multiple base dimensions of the same type.

Fields¶

Name	Type	Default
`geography`	`str` \| None	`None`
`metric`	`str` \| None	`None`
`model_year`	`str` \| None	`None`
`scenario`	`str` \| None	`None`
`sector`	`str` \| None	`None`
`subsector`	`str` \| None	`None`
`time`	`str` \| None	`None`
`weather_year`	`str` \| None	`None`

DaylightSavingAdjustmentModel¶

dsgrid.dimension.time.DaylightSavingAdjustmentModel

Defines how to drop and add data along with timestamps to convert standard time load profiles to clock time

Fields¶

Name	Type	Default	Description
`spring_forward_hour`	DaylightSavingSpringForwardType	`DaylightSavingSpringForwardType.NONE`	Data adjustment for spring forward hour (a 2AM in March)
`fall_back_hour`	DaylightSavingFallBackType	`DaylightSavingFallBackType.NONE`	Data adjustment for spring forward hour (a 2AM in November)

DimensionMappingsModel¶

dsgrid.config.project_config.DimensionMappingsModel

Defines all dimension mappings associated with a dsgrid project, including base-to-supplemental mappings and dataset-to-project mappings.

Fields¶

Name	Type	Default	Description
`base_to_supplemental_references`	list[DimensionMappingReferenceModel]	`[]`	Base dimension to supplemental dimension mappings (e.g., county-to-state) used to support various queries and dimension transformations.
`dataset_to_project`	dict[`str`, list[DimensionMappingReferenceModel]]	`{}`	Dataset-to-project mappings map dataset dimensions to project dimensions. Once a dataset is submitted to a project, dsgrid adds the dataset-to-project mappings to the project config. Some projects may not have any dataset-to-project mappings. Dataset-to-project mappings are only supplied if a dataset’s dimensions do not match the project’s dimension.

DimensionsModel¶

dsgrid.config.project_config.DimensionsModel

Contains dimensions defined by a project

Fields¶

Name	Type	Default	Description
`base_dimensions`	list[DimensionModel \| DateTimeDimensionModel \| AnnualTimeDimensionModel \| RepresentativePeriodTimeDimensionModel \| DatetimeExternalTimeZoneDimensionModel \| IndexTimeDimensionModel \| NoOpTimeDimensionModel]	`[]`	List of dimensions for a project’s base dimensions. They will be automatically registered during project registration and then converted to base_dimension_references.
`base_dimension_references`	list[DimensionReferenceModel]	`[]`	List of registry references (`DimensionReferenceModel`) for a project’s base dimensions.
`subset_dimensions`	list[SubsetDimensionGroupModel]	`[]`	List of subset dimension groups. Subset dimension groups are used to specify subsets of base dimension records that a dataset must support, dimensionality of derived datasets, and query filters. Subset dimension groups also define a new supplemental dimension whose records correspond to the table columns/subset selectors, such that defining a subset dimension group can be a convenient way to define reporting at a different level of aggregation as compared to the project’s base dimensions.
`supplemental_dimensions`	list[SupplementalDimensionModel]	`[]`	List of supplemental dimensions. They will be automatically registered. during project registration and then converted to supplemental_dimension_references. Supplemental dimensions are used to support additional querying and transformations (e.g., aggregations, disgaggregations, filtering, scaling, etc.) of the project’s base data.
`supplemental_dimension_references`	list[DimensionReferenceModel]	`[]`	List of registry references for a project’s supplemental dimensions. Dimensions references of the same :class:`dsgrid.dimensions.base_model.DimensionType` are allowed for supplemental dimension references (i.e., multiple `Geography` types are allowed).

Validators¶

Name	Applies To	Description
`check_files`	`check_files`	Validate dimension files are unique across all dimensions
`check_names`	`check_names`	Validate dimension names are unique across all dimensions.
`check_time_zone`	`check_time_zone`	Validate the time zone column in geography records.
`check_subset_dimensions`	`check_subset_dimensions`	Check that each subset dimension has a unique name.
`check_dimensions`	`(model)`	Validate that the dimensions are complete and consistent.
`pre_check_values`	`(model)`	Checks that base dimensions are defined.
`check_dimension_names`	`(model)`	Check that all dimension query names are unique.

InputDatasetModel¶

dsgrid.config.project_config.InputDatasetModel

Defines an input dataset for the project config.

Fields¶

Name	Type	Default	Description
`dataset_id`	`str`	(required)	Unique dataset identifier.
`dataset_type`	InputDatasetType	(required)	Dataset type.
`version`	`str` \| None	`None`	Version of the registered dataset. The version specification is optional. If no version is supplied, then the latest version in the registry is assumed. The version string must be in semver format (e.g., ‘1.0.0’) and it must be a valid/existing version in the registry.
`required_dimensions`	RequiredDimensionsModel	single_dimensional=RequiredDimensionRecordsModel(geography=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), metric=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), model_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), scenario=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), sector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), subsector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]), weather_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=['__all__'], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])) multi_dimensional=[]	Defines required record IDs that must exist for each dimension.
`mapping_references`	list[DimensionMappingReferenceModel]	`[]`	Defines how to map the dataset dimensions to the project. Auto-populated during submission.
`base_dimension_names`	DatasetBaseDimensionNamesModel	`geography=None metric=None model_year=None scenario=None sector=None subsector=None time=None weather_year=None`	Defines the project base dimensions to which the dataset will map itself. Auto-populated during submission.
`status`	DatasetRegistryStatus	`"Unregistered"`	Registration status of the dataset, added by dsgrid.
`wrap_time_allowed`	`bool`	`False`	Whether to allow dataset time to be wrapped to project time if different
`time_based_data_adjustment`	TimeBasedDataAdjustmentModel	`leap_day_adjustment=<LeapDayAdjustmentType.NONE: 'none'> daylight_saving_adjustment=DaylightSavingAdjustmentModel(spring_forward_hour=<DaylightSavingSpringForwardType.NONE: 'none'>, fall_back_hour=<DaylightSavingFallBackType.NONE: 'none'>)`	Defines how the rest of the dataframe is adjusted with respect to time. E.g., when drop associated data when dropping a leap day timestamp.

Validators¶

Name	Applies To	Description
`check_data_adjustment`	`check_data_adjustment`	Check daylight saving adjustment

MappingTableByNameModel¶

dsgrid.config.mapping_tables.MappingTableByNameModel

Attributes for a dimension mapping table for soon-to-be registered dimensions by name. This will be converted to a MappingTableModel as soon as the dimensions are registered.

Fields¶

Name	Type	Default	Description
`mapping_type`	DimensionMappingType	`"many_to_one_aggregation"`	Type/purpose of the dimension mapping
`archetype`	DimensionMappingArchetype \| None	`None`	Dimension mapping archetype, determined based on mapping_type
`description`	`str` \| None	`None`	Description of dimension mapping
`from_fraction_tolerance`	`float`	`1e-06`	Tolerance value to apply to the from_fraction column
`to_fraction_tolerance`	`float`	`1e-06`	Tolerance value to apply to the to_fraction column
`project_base_dimension_name`	`str` \| None	`None`	Name of the base dimension for which the mapping is being registered. This is required in cases where the project has multiple base dimensions of the same type. If None, there must only be one base dimension of this type in the project.
`filename`	`str`	(required)	Filename containing association table records.

RequiredBaseDimensionModel¶

dsgrid.config.project_config.RequiredBaseDimensionModel

Base data model for all dsgrid data models

Fields¶

Name	Type	Default	Description
`record_ids`	list[`str`]	`[]`
`dimension_name`	`str` \| None	`None`	Identifies which base dimension contains the record IDs. Required if there is more than one base dimension for a given dimension type.

RequiredDimensionRecordsByTypeModel¶

dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel

Base data model for all dsgrid data models

Fields¶

Name	Type	Default
`base`	RequiredBaseDimensionModel	`record_ids=[] dimension_name=None`
`base_missing`	RequiredBaseDimensionModel	`record_ids=[] dimension_name=None`
`subset`	list[RequiredSubsetDimensionRecordsModel]	`[]`

Validators¶

Name	Applies To	Description
`handle_legacy_format`	`(model)`	No description
`check_base`	`(model)`	No description

RequiredDimensionRecordsModel¶

dsgrid.config.project_config.RequiredDimensionRecordsModel

Base data model for all dsgrid data models

Fields¶

Name	Type	Default
`geography`	RequiredDimensionRecordsByTypeModel	`base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) subset=[]`
`metric`	RequiredDimensionRecordsByTypeModel	`base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) subset=[]`
`model_year`	RequiredDimensionRecordsByTypeModel	`base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) subset=[]`
`scenario`	RequiredDimensionRecordsByTypeModel	`base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) subset=[]`
`sector`	RequiredDimensionRecordsByTypeModel	`base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) subset=[]`
`subsector`	RequiredDimensionRecordsByTypeModel	`base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) subset=[]`
`weather_year`	RequiredDimensionRecordsByTypeModel	`base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None) subset=[]`

RequiredDimensionsModel¶

dsgrid.config.project_config.RequiredDimensionsModel

Defines required record IDs that must exist for each dimension in a dataset. Record IDs can reside in the project’s base or subset dimensions.

Requirements can be specified for a single dimension or a combination of dimensions. For example, if a project includes commercial, residential, and transportation sectors but the dataset has only transporation sector records, it should specify a single_dimensional requirement that is a subset of of the project’s base dimension.

{"single_dimensional": "sector": {"base": {"record_ids": ["transportation"]}}}

If a dataset’s requirements span multiple dimensions, such as if it does not have some metric records for some geography records, then a multi_dimensional requirement should be specified. (By default, a full cross join is assumed to be present.)

{"multi_dimensional": {
    "geography": {"base": {"record_ids": ["12345"]}}
    "metric": {"base": {"record_ids": ["electricity_cooling"]}}
  }
}

If a dataset specifies a dimension type within a multi_dimensional section and wants to use all records from a project base dimension, it can specify base.record_ids = ["__all__"] as a shorthand notation.

Requirements for a dimension cannot be defined in both single_dimensional and multi_dimensional sections.

If no records are listed for a dimension then all project base records are required.

It might be easier for a dataset to specify what it does not have rather than what it does have. In that case, it is recommended to use the RequiredDimensionRecordsModel.base_missing field. dsgrid will compute the difference of the base dimension records and the base_missing records to determine the dataset’s required records.

If a project has multiple base dimensions of the same type, the RequiredDimensionRecordsModel.dimension_name must be specified to identify the base dimension that contains the record IDs.

If a dataset contains a subset of project base dimension records that are defined in the project’s subset dimensions, it is recommended to use that specification. dsgrid will substitute base records for mapped subset records at runtime.

Fields¶

Name	Type	Default	Description
`single_dimensional`	RequiredDimensionRecordsModel	geography=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]) metric=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]) model_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]) scenario=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]) sector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]) subsector=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[]) weather_year=RequiredDimensionRecordsByTypeModel(base=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), base_missing=RequiredBaseDimensionModel(record_ids=[], dimension_name=None), subset=[])	Required records for a single dimension.
`multi_dimensional`	list[RequiredDimensionRecordsModel]	`[]`	Required records for a combination of dimensions. For example, there may be a dataset requirement for only one subsector for a given sector instead of a cross product.

Validators¶

Name	Applies To	Description
`check_for_duplicates`	`(model)`	1. Ensure that the same dimension does not have requirements in both single and multi

RequiredSubsetDimensionRecordsModel¶

dsgrid.config.project_config.RequiredSubsetDimensionRecordsModel

Base data model for all dsgrid data models

Fields¶

Name	Type	Default	Description
`name`	`str`	(required)	Name of a subset dimension
`selectors`	list[`str`]	(required)	One or more selectors in the subset dimension

SubsetDimensionGroupModel¶

dsgrid.config.project_config.SubsetDimensionGroupModel

Defines one or more subset dimension selectors for a dimension type.

Fields¶

Name	Type	Default	Description
`name`	`str`	(required)
`description`	`str`	(required)
`dimension_type`	DimensionType	(required)	Type of the dimension
`filename`	`str` \| None	`None`	Filename containing dimension records. Only populated for initial registration. Each selector’s records are stored as JSON objects in the dsgrid registry.
`selectors`	list[SubsetDimensionSelectorModel]	(required)	Dimension selectors
`selector_references`	list[DimensionReferenceModel]	`[]`	References to the subset dimensions generated by dsgrid during registration.
`create_supplemental_dimension`	`bool`	`True`	Auto-generate supplemental dimensions in order to allow aggregrations on the subsets.
`base_dimension_name`	`str` \| None	`None`	Name of base dimension for the supplemental dimension mapping, if create_supplemental_dimension is true. Required if there are multiple base dimensions for this type.
`record_ids`	`set`	`set()`

Validators¶

Name	Applies To	Description
`check_selectors`	`check_selectors`	Check that the selectors are defined consistently.
`load_records`	`(model)`	Load the records for each subset dimension selector.

SubsetDimensionSelectorModel¶

dsgrid.config.project_config.SubsetDimensionSelectorModel

Defines a subset dimension selector inclusive of the subset’s records and information required to define the selector as a record within the supplemental dimension defined by the subset dimension group.

Fields¶

Name	Type	Default	Description
`name`	`str`	(required)
`description`	`str`	(required)
`column_values`	dict[`str`, `str`]	`{}`	Optional columns to populate in the subset dimension group’s supplemental dimension records table. For example, if each selector in the group defines the end uses for one sector (e.g., commercial_end_uses, transportation_end_uses), the supplemental dimension records table needs to define the ‘fuel_id’ and ‘unit’ fields of the EnergyEndUse data model.
`records`	list[`str`]	`[]`	Table of values populated by reading the parent subset dimension records file. Should not be populated by the user.

SupplementalDimensionModel¶

dsgrid.config.supplemental_dimension.SupplementalDimensionModel

Defines a supplemental dimension.

Fields¶

Name	Type	Default	Description
`id`	`int` \| None	`None`	Registry database ID
`version`	`str` \| None	`None`	Version, generated by dsgrid
`name`	`str`	(required)	Dimension name
`dimension_type`	DimensionType	(required)	Type of the dimension
`dimension_id`	`str` \| None	`None`	Unique identifier, generated by dsgrid
`module`	`str`	`"dsgrid.dimension.standard"`	Python module with the dimension class
`class_name`	`str`	(required)	Dimension record model class name. The dimension class defines the expected and allowable fields (and their data types) for the dimension records file.All dimension records must have a ‘id’ and ‘name’ field.Some dimension classes support additional fields that can be used for mapping, querying, display, etc.dsgrid in online-mode only supports dimension classes defined in the :mod:`dsgrid.dimension.standard` module. If dsgrid does not currently support a dimension class that you require, please contact the dsgrid-coordination team to request a new class feature
`cls`	`Any`	`None`	Dimension record model class
`description`	`str` \| None	`None`	A description of the dimension records that is helpful, memorable, and identifiable
`filename`	`str` \| None	`None`	Filename containing dimension records. Only assigned for user input and output purposes. The registry database stores records in the dimension JSON document.
`file_hash`	`str` \| None	`None`	Hash of the contents of the file
`records`	`list`	`[]`	Dimension records that can either be loaded from filename at runtime or provided directly. Example of records provided directly: records: [ {id: ‘scenario_1’, name: ‘Scenario 1’}, {id: ‘scenario_2’, name: ‘Scenario 2’}, ],
`mapping`	MappingTableByNameModel	(required)	Defines how the supplemental dimension will be mapped to the project’s base dimension.

Validators¶

Name	Applies To	Description
`check_name`	`check_name`	No description
`check_module`	`check_module`	No description
`get_dimension_class_name`	`get_dimension_class_name`	Set class_name based on inputs.
`get_dimension_class`	`get_dimension_class`	No description
`check_file`	`check_file`	Validate that dimension file exists and has no errors
`compute_file_hash`	`compute_file_hash`	No description
`add_records`	`add_records`	Add records from the file.

TimeBasedDataAdjustmentModel¶

dsgrid.dimension.time.TimeBasedDataAdjustmentModel

Defines how data needs to be adjusted with respect to time. For leap day adjustment, up to one full day of timestamps and data are dropped. For daylight savings, the dataframe is adjusted alongside the timestamps. This is useful when the load profiles are modeled in standard time and need to be converted to get clock time load profiles.

Fields¶

Name	Type	Default	Description
`leap_day_adjustment`	LeapDayAdjustmentType	`LeapDayAdjustmentType.NONE`	Leap day adjustment method applied to time data. The dsgrid default is None, i.e., no adjustment made to leap years. Adjustments are made to leap years only.
`daylight_saving_adjustment`	DaylightSavingAdjustmentModel	`spring_forward_hour=<DaylightSavingSpringForwardType.NONE: 'none'> fall_back_hour=<DaylightSavingFallBackType.NONE: 'none'>`	Daylight saving adjustment method applied to time data