Project Config

pydantic model dsgrid.config.project_config.ProjectConfigModel

Represents project configurations

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• datasets (list[dsgrid.config.project_config.InputDatasetModel])

• description (str)

• dimension_mappings (dsgrid.config.project_config.DimensionMappingsModel)

• dimensions (dsgrid.config.project_config.DimensionsModel)

• id (str | None)

• key (str | None)

• name (str)

• project_id (str)

• rev (str | None)

• status (dsgrid.registry.common.ProjectRegistryStatus)

• version (str | None)

Validators:

• check_project_id_handle » project_id

field datasets: list[InputDatasetModel] [Required]

List of input datasets for the project.

field description: str [Required]

Detailed project description.

field dimension_mappings: DimensionMappingsModel = DimensionMappingsModel(base_to_supplemental_references=[], dataset_to_project={})

List of project mappings. Initialized with base-to-base and base-to-supplemental mappings. dataset-to-project mappings are added by dsgrid as datasets get registered with the project.

field dimensions: DimensionsModel [Required]

List of base and supplemental dimensions.

field id: str | None = None (alias '_id')

Registry database ID

field key: str | None = None (alias '_key')

Registry database key

field name: str [Required]

A project name to accompany the ID.

field project_id: str [Required]

A unique project identifier that is project-specific (e.g., ‘standard-scenarios-2021’).

Validated by:

• check_project_id_handle

field rev: str | None = None (alias '_rev')

Registry database revision

field status: ProjectRegistryStatus = ProjectRegistryStatus.INITIAL_REGISTRATION

project registry status

field version: str | None = None

Version, generated by dsgrid

validator check_project_id_handle  »  project_id

Check for valid characters in project id

pydantic model dsgrid.config.project_config.InputDatasetModel

Defines an input dataset for the project config.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• dataset_id (str)

• dataset_type (dsgrid.config.dataset_config.InputDatasetType)

• mapping_references (list[dsgrid.config.dimension_mapping_base.DimensionMappingReferenceModel])

• required_dimensions (dsgrid.config.project_config.RequiredDimensionsModel)

• status (dsgrid.registry.common.DatasetRegistryStatus)

• time_based_data_adjustment (dsgrid.dimension.time.TimeBasedDataAdjustmentModel)

• version (str)

• wrap_time_allowed (bool)

Validators:

• check_data_adjustment » time_based_data_adjustment

field dataset_id: str [Required]

Unique dataset identifier.

field dataset_type: InputDatasetType [Required]

Dataset type.

field mapping_references: list[DimensionMappingReferenceModel] = []

Defines how to map the dataset dimensions to the project.

field required_dimensions: RequiredDimensionsModel = RequiredDimensionsModel(single_dimensional=RequiredDimensionRecordsModel(geography=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), metric=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), model_year=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), scenario=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), sector=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), subsector=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), weather_year=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])), multi_dimensional=[])

Defines required record IDs that must exist for each dimension.

field status: DatasetRegistryStatus = DatasetRegistryStatus.UNREGISTERED

Registration status of the dataset, added by dsgrid.

field time_based_data_adjustment: TimeBasedDataAdjustmentModel = TimeBasedDataAdjustmentModel(leap_day_adjustment=<LeapDayAdjustmentType.NONE: 'none'>, daylight_saving_adjustment=DaylightSavingAdjustmentModel(spring_forward_hour=<DaylightSavingSpringForwardType.NONE: 'none'>, fall_back_hour=<DaylightSavingFallBackType.NONE: 'none'>))

Defines how the rest of the dataframe is adjusted with respect to time. E.g., when drop associated data when dropping a leap day timestamp.

Validated by:

• check_data_adjustment

field version: str = None

Version of the registered dataset

field wrap_time_allowed: bool = False

Whether to allow dataset time to be wrapped to project time if different

validator check_data_adjustment  »  time_based_data_adjustment

Check daylight saving adjustment

pydantic model dsgrid.config.project_config.DimensionsModel

Contains dimensions defined by a project

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base_dimension_references (list[dsgrid.config.dimensions.DimensionReferenceModel])

• base_dimensions (list[dsgrid.config.dimensions.DimensionModel | dsgrid.config.dimensions.DateTimeDimensionModel | dsgrid.config.dimensions.AnnualTimeDimensionModel | dsgrid.config.dimensions.RepresentativePeriodTimeDimensionModel | dsgrid.config.dimensions.IndexTimeDimensionModel | dsgrid.config.dimensions.NoOpTimeDimensionModel])

• subset_dimensions (list[dsgrid.config.project_config.SubsetDimensionGroupModel])

• supplemental_dimension_references (list[dsgrid.config.dimensions.DimensionReferenceModel])

• supplemental_dimensions (list[dsgrid.config.supplemental_dimension.SupplementalDimensionModel])

Validators:

• check_dimension_query_names » all fields

• check_dimensions » all fields

• check_files » base_dimensions

• check_names » base_dimensions

• check_subset_dimensions » subset_dimensions

• check_time_zone » base_dimensions

• pre_check_values » all fields

field base_dimension_references: list[DimensionReferenceModel] = []

List of registry references (DimensionReferenceModel) for a project’s base dimensions.

Validated by:

• check_dimension_query_names

• check_dimensions

• pre_check_values

field base_dimensions: Annotated[list[DimensionModel | DateTimeDimensionModel | AnnualTimeDimensionModel | RepresentativePeriodTimeDimensionModel | IndexTimeDimensionModel | NoOpTimeDimensionModel], BeforeValidator(func=handle_dimension_union, json_schema_input_type=PydanticUndefined)] = []

List of dimensions for a project’s base dimensions. They will be automatically registered during project registration and then converted to base_dimension_references.

Constraints:

• func = <function handle_dimension_union at 0x7fdfb5e6f380>

• json_schema_input_type = PydanticUndefined

Validated by:

• check_dimension_query_names

• check_dimensions

• check_files

• check_names

• check_time_zone

• pre_check_values

field subset_dimensions: list[SubsetDimensionGroupModel] = []

List of subset dimension groups

Validated by:

• check_dimension_query_names

• check_dimensions

• check_subset_dimensions

• pre_check_values

field supplemental_dimension_references: list[DimensionReferenceModel] = []

List of registry references for a project’s supplemental dimensions.

Validated by:

• check_dimension_query_names

• check_dimensions

• pre_check_values

field supplemental_dimensions: list[SupplementalDimensionModel] = []

List of supplemental dimensions. They will be automatically registered during project registration and then converted to supplemental_dimension_references.

Validated by:

• check_dimension_query_names

• check_dimensions

• pre_check_values

validator check_dimension_query_names  »  all fields

Check that all dimension query names are unique.

validator check_dimensions  »  all fields

Validate that the dimensions are complete and consistent.

validator check_files  »  base_dimensions

Validate dimension files are unique across all dimensions

validator check_names  »  base_dimensions

Validate dimension names are unique across all dimensions.

validator check_subset_dimensions  »  subset_dimensions

Check that each subset dimension has a unique name and display_name.

validator check_time_zone  »  base_dimensions

Validate the time zone column in geography records.

validator pre_check_values  »  all fields

Checks that base dimensions are defined.

pydantic model dsgrid.config.project_config.DimensionReferenceModel

Reference to a dimension stored in the registry

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• dimension_id (str)

• dimension_type (dsgrid.dimension.base_models.DimensionType)

• version (str)

field dimension_id: str [Required]

Unique ID of the dimension in the registry

field dimension_type: DimensionType [Required] (alias 'type')

Type of the dimension

field version: str [Required]

Version of the dimension

pydantic model dsgrid.config.project_config.RequiredDimensionsModel

Defines required record IDs that must exist for each dimension in a dataset. Record IDs can reside in the project’s base, subset, or supplemental dimensions. Using subset dimensions is recommended. dsgrid will substitute base records for mapped subset records at runtime. If no records are listed for a dimension then all project base records are required.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• multi_dimensional (list[dsgrid.config.project_config.RequiredDimensionRecordsModel])

• single_dimensional (dsgrid.config.project_config.RequiredDimensionRecordsModel)

Validators:

• check_for_duplicates » all fields

field multi_dimensional: list[RequiredDimensionRecordsModel] = []

Required records for a combination of dimensions. For example, there may be a dataset requirement for only one subsector for a given sector instead of a cross product.

Validated by:

• check_for_duplicates

field single_dimensional: RequiredDimensionRecordsModel = RequiredDimensionRecordsModel(geography=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), metric=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), model_year=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), scenario=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), sector=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), subsector=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]), weather_year=RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[]))

Required records for a single dimension.

Validated by:

• check_for_duplicates

validator check_for_duplicates  »  all fields

pydantic model dsgrid.config.project_config.DimensionMappingsModel

Defines all dimension mappings associated with a dsgrid project, including base-to-supplemental mappings and dataset-to-project mappings.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base_to_supplemental_references (list[dsgrid.config.dimension_mapping_base.DimensionMappingReferenceModel])

• dataset_to_project (dict[str, list[dsgrid.config.dimension_mapping_base.DimensionMappingReferenceModel]])

field base_to_supplemental_references: list[DimensionMappingReferenceModel] = []

Base dimension to supplemental dimension mappings (e.g., county-to-state) used to support various queries and dimension transformations.

field dataset_to_project: dict[str, list[DimensionMappingReferenceModel]] = {}

Dataset-to-project mappings map dataset dimensions to project dimensions.

pydantic model dsgrid.config.project_config.RequiredDimensionRecordsModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• geography (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• metric (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• model_year (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• scenario (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• sector (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• subsector (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

• weather_year (dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel)

field geography: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])

field metric: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])

field model_year: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])

field scenario: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])

field sector: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])

field subsector: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])

field weather_year: RequiredDimensionRecordsByTypeModel = RequiredDimensionRecordsByTypeModel(base=[], base_missing=[], subset=[], supplemental=[])

pydantic model dsgrid.config.project_config.RequiredSupplementalDimensionRecordsModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• name (str)

• record_ids (list[str])

field name: str [Required]

Name of a supplemental dimension

field record_ids: list[str] [Required]

One or more record IDs in the supplemental dimension

pydantic model dsgrid.config.project_config.RequiredDimensionRecordsByTypeModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• base (list[str])

• base_missing (list[str])

• subset (list[dsgrid.config.project_config.RequiredSubsetDimensionRecordsModel])

• supplemental (list[dsgrid.config.project_config.RequiredSupplementalDimensionRecordsModel])

Validators:

• check_base » all fields

field base: list[str] = []

Validated by:

• check_base

field base_missing: list[str] = []

Validated by:

• check_base

field subset: list[RequiredSubsetDimensionRecordsModel] = []

Validated by:

• check_base

field supplemental: list[RequiredSupplementalDimensionRecordsModel] = []

Validated by:

• check_base

validator check_base  »  all fields

defines_dimension_requirement() → bool

Returns True if the model defines a dimension requirement.

pydantic model dsgrid.config.project_config.SubsetDimensionGroupModel

Defines one or more subset dimension selectors for a dimension type.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• create_supplemental_dimension (bool)

• description (str)

• dimension_query_name (str | None)

• dimension_type (dsgrid.dimension.base_models.DimensionType)

• display_name (str)

• filename (str | None)

• name (str)

• record_ids (set[str])

• selector_references (list[dsgrid.config.dimensions.DimensionReferenceModel])

• selectors (list[dsgrid.config.project_config.SubsetDimensionSelectorModel])

Validators:

• check_display_name » display_name

• check_query_name » dimension_query_name

• check_selectors » selectors

• load_records » all fields

field create_supplemental_dimension: bool = True

Auto-generate supplemental dimensions in order to allow aggregrations on the subsets.

Validated by:

• load_records

field description: str [Required]

Validated by:

• load_records

field dimension_query_name: str | None = None

Auto-generated query name for SQL queries.

Validated by:

• check_query_name

• load_records

field dimension_type: DimensionType [Required] (alias 'type')

Type of the dimension

Validated by:

• load_records

field display_name: str [Required]

Validated by:

• check_display_name

• load_records

field filename: str | None = None (alias 'file')

Filename containing dimension records. Only populated for initial registration. Each selector’s records are stored as JSON objects in the dsgrid registry.

Validated by:

• load_records

field name: str [Required]

Validated by:

• load_records

field record_ids: set[str] = {}

Validated by:

• load_records

field selector_references: list[DimensionReferenceModel] = []

References to the subset dimensions generated by dsgrid during registration.

Validated by:

• load_records

field selectors: list[SubsetDimensionSelectorModel] [Required]

Dimension selectors

Validated by:

• check_selectors

• load_records

validator check_display_name  »  display_name

validator check_query_name  »  dimension_query_name

validator check_selectors  »  selectors

Check that the selectors are defined consistently.

validator load_records  »  all fields

Load the records for each subset dimension selector.

pydantic model dsgrid.config.project_config.SubsetDimensionGroupListModel

Defines a list of subset dimensions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• subset_dimensions (List[dsgrid.config.project_config.SubsetDimensionGroupModel])

field subset_dimensions: Annotated[List[SubsetDimensionGroupModel], Len(min_length=1, max_length=None)] [Required]

List of subset dimensions to be registered

Constraints:

• min_length = 1

To add new supplemental dimensions

pydantic model dsgrid.config.supplemental_dimension.SupplementalDimensionsListModel

Defines a list of supplemental dimensions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• supplemental_dimensions (List[dsgrid.config.supplemental_dimension.SupplementalDimensionModel])

field supplemental_dimensions: Annotated[List[SupplementalDimensionModel], Len(min_length=1, max_length=None)] [Required]

List of supplemental dimensions and mappings to be registered

Constraints:

• min_length = 1

To add new dataset requirements

pydantic model dsgrid.config.input_dataset_requirements.InputDatasetListModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• datasets (List[dsgrid.config.project_config.InputDatasetModel])

field datasets: Annotated[List[InputDatasetModel], Len(min_length=1, max_length=None)] [Required]

List of input datasets for the project.

Constraints:

• min_length = 1

To replace dataset dimension requirements

pydantic model dsgrid.config.input_dataset_requirements.InputDatasetDimensionRequirementsListModel

Defines a list of dataset dimension requirements.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• dataset_dimension_requirements (List[dsgrid.config.input_dataset_requirements.InputDatasetDimensionRequirementsModel])

field dataset_dimension_requirements: Annotated[List[InputDatasetDimensionRequirementsModel], Len(min_length=1, max_length=None)] [Required]

List of dataset dimension requirements

Constraints:

• min_length = 1

pydantic model dsgrid.config.input_dataset_requirements.InputDatasetDimensionRequirementsModel

Defines dataset dimension requirements.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

• dataset_id (str)

• required_dimensions (dsgrid.config.project_config.RequiredDimensionsModel)

field dataset_id: str [Required]

field required_dimensions: RequiredDimensionsModel [Required]

Defines required record IDs that must exist for each dimension.