Dataset Config

pydantic model dsgrid.config.dataset_config.DatasetConfigModel[source]

Represents dataset configurations.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

__init__ uses __pydantic_self__ instead of the more common self for the first arg to allow self as a field name.

digraph "Entity Relationship Diagram created by erdantic" {
   graph [fontcolor=gray66,
      fontname="Times New Roman,Times,Liberation Serif,serif",
      fontsize=9,
      nodesep=0.5,
      rankdir=LR,
      ranksep=1.5
   ];
   node [fontname="Times New Roman,Times,Liberation Serif,serif",
      fontsize=14,
      label="\N",
      shape=plain
   ];
   edge [dir=both];
   "dsgrid.config.dataset_config.DatasetConfigModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>DatasetConfigModel</b></td></tr><tr><td>dataset_id</td><td port="dataset_id">str</td></tr><tr><td>version</td><td port="version">Optional[str]</td></tr><tr><td>dataset_type</td><td port="dataset_type">InputDatasetType</td></tr><tr><td>dataset_qualifier_metadata</td><td port="dataset_qualifier_metadata">Optional[Union[QuantityModel, GrowthRateModel]]</td></tr><tr><td>sector_description</td><td port="sector_description">Optional[str]</td></tr><tr><td>data_source</td><td port="data_source">str</td></tr><tr><td>data_schema</td><td port="data_schema">Union[StandardDataSchemaModel, OneTableDataSchemaModel]</td></tr><tr><td>dataset_version</td><td port="dataset_version">Optional[str]</td></tr><tr><td>description</td><td port="description">str</td></tr><tr><td>origin_creator</td><td port="origin_creator">str</td></tr><tr><td>origin_organization</td><td port="origin_organization">str</td></tr><tr><td>origin_contributors</td><td port="origin_contributors">List[str]</td></tr><tr><td>origin_project</td><td port="origin_project">str</td></tr><tr><td>origin_date</td><td port="origin_date">str</td></tr><tr><td>origin_version</td><td port="origin_version">str</td></tr><tr><td>source</td><td port="source">str</td></tr><tr><td>data_classification</td><td port="data_classification">DataClassificationType</td></tr><tr><td>tags</td><td port="tags">List[str]</td></tr><tr><td>enable_unit_conversion</td><td port="enable_unit_conversion">bool</td></tr><tr><td>use_project_geography_time_zone</td><td port="use_project_geography_time_zone">bool</td></tr><tr><td>dimensions</td><td port="dimensions">List[Union[DimensionModel, DateTimeDimensionModel, AnnualTimeDimensionModel, RepresentativePeriodTimeDimensionModel, NoOpTimeDimensionModel]]</td></tr><tr><td>dimension_references</td><td port="dimension_references">List[DimensionReferenceModel]</td></tr><tr><td>user_defined_metadata</td><td port="user_defined_metadata">dict[]</td></tr><tr><td>trivial_dimensions</td><td port="trivial_dimensions">Optional[List[DimensionType]]</td></tr><tr><td>id</td><td port="id">Optional[str]</td></tr><tr><td>key</td><td port="key">Optional[str]</td></tr><tr><td>rev</td><td port="rev">Optional[str]</td></tr></table>>,
      tooltip="dsgrid.config.dataset_config.DatasetConfigModel&#xA;&#xA;Represents dataset configurations.&#xA;"];
   "dsgrid.config.dataset_config.GrowthRateModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>GrowthRateModel</b></td></tr><tr><td>dataset_qualifier_type</td><td port="dataset_qualifier_type">Literal[DatasetQualifierType.GROWTH_RATE]</td></tr><tr><td>growth_rate_type</td><td port="growth_rate_type">GrowthRateType</td></tr></table>>,
      tooltip="dsgrid.config.dataset_config.GrowthRateModel&#xA;&#xA;Base data model for all dsgrid data models&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dataset_qualifier_metadata:e -> "dsgrid.config.dataset_config.GrowthRateModel":_root:w   [arrowhead=noneteeodot,
      arrowtail=nonenone];
   "dsgrid.config.dataset_config.OneTableDataSchemaModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>OneTableDataSchemaModel</b></td></tr><tr><td>data_schema_type</td><td port="data_schema_type">Literal[DataSchemaType.ONE_TABLE]</td></tr><tr><td>table_format</td><td port="table_format">Union[PivotedTableFormatModel, UnpivotedTableFormatModel]</td></tr></table>>,
      tooltip="dsgrid.config.dataset_config.OneTableDataSchemaModel&#xA;&#xA;Base data model for all dsgrid data models&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":data_schema:e -> "dsgrid.config.dataset_config.OneTableDataSchemaModel":_root:w   [arrowhead=noneteetee,
      arrowtail=nonenone];
   "dsgrid.config.dataset_config.QuantityModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>QuantityModel</b></td></tr><tr><td>dataset_qualifier_type</td><td port="dataset_qualifier_type">Literal[DatasetQualifierType.QUANTITY]</td></tr></table>>,
      tooltip="dsgrid.config.dataset_config.QuantityModel&#xA;&#xA;Base data model for all dsgrid data models&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dataset_qualifier_metadata:e -> "dsgrid.config.dataset_config.QuantityModel":_root:w   [arrowhead=noneteeodot,
      arrowtail=nonenone];
   "dsgrid.config.dataset_config.StandardDataSchemaModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>StandardDataSchemaModel</b></td></tr><tr><td>data_schema_type</td><td port="data_schema_type">Literal[DataSchemaType.STANDARD]</td></tr><tr><td>table_format</td><td port="table_format">Union[PivotedTableFormatModel, UnpivotedTableFormatModel]</td></tr></table>>,
      tooltip="dsgrid.config.dataset_config.StandardDataSchemaModel&#xA;&#xA;Base data model for all dsgrid data models&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":data_schema:e -> "dsgrid.config.dataset_config.StandardDataSchemaModel":_root:w   [arrowhead=noneteetee,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.AnnualTimeDimensionModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>AnnualTimeDimensionModel</b></td></tr><tr><td>name</td><td port="name">str</td></tr><tr><td>display_name</td><td port="display_name">str</td></tr><tr><td>dimension_query_name</td><td port="dimension_query_name">Optional[str]</td></tr><tr><td>dimension_type</td><td port="dimension_type">DimensionType</td></tr><tr><td>dimension_id</td><td port="dimension_id">Optional[str]</td></tr><tr><td>version</td><td port="version">Optional[str]</td></tr><tr><td>module</td><td port="module">Optional[str]</td></tr><tr><td>class_name</td><td port="class_name">str</td></tr><tr><td>cls</td><td port="cls">Optional[Any]</td></tr><tr><td>description</td><td port="description">str</td></tr><tr><td>id</td><td port="id">Optional[str]</td></tr><tr><td>key</td><td port="key">Optional[str]</td></tr><tr><td>rev</td><td port="rev">Optional[str]</td></tr><tr><td>time_type</td><td port="time_type">TimeDimensionType</td></tr><tr><td>measurement_type</td><td port="measurement_type">MeasurementType</td></tr><tr><td>str_format</td><td port="str_format">Optional[str]</td></tr><tr><td>ranges</td><td port="ranges">List[TimeRangeModel]</td></tr><tr><td>include_leap_day</td><td port="include_leap_day">bool</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.AnnualTimeDimensionModel&#xA;&#xA;Defines an annual time dimension where timestamps are years.&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dimensions:e -> "dsgrid.config.dimensions.AnnualTimeDimensionModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.DateTimeDimensionModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>DateTimeDimensionModel</b></td></tr><tr><td>name</td><td port="name">str</td></tr><tr><td>display_name</td><td port="display_name">str</td></tr><tr><td>dimension_query_name</td><td port="dimension_query_name">Optional[str]</td></tr><tr><td>dimension_type</td><td port="dimension_type">DimensionType</td></tr><tr><td>dimension_id</td><td port="dimension_id">Optional[str]</td></tr><tr><td>version</td><td port="version">Optional[str]</td></tr><tr><td>module</td><td port="module">Optional[str]</td></tr><tr><td>class_name</td><td port="class_name">str</td></tr><tr><td>cls</td><td port="cls">Optional[Any]</td></tr><tr><td>description</td><td port="description">str</td></tr><tr><td>id</td><td port="id">Optional[str]</td></tr><tr><td>key</td><td port="key">Optional[str]</td></tr><tr><td>rev</td><td port="rev">Optional[str]</td></tr><tr><td>time_type</td><td port="time_type">TimeDimensionType</td></tr><tr><td>measurement_type</td><td port="measurement_type">MeasurementType</td></tr><tr><td>str_format</td><td port="str_format">Optional[str]</td></tr><tr><td>frequency</td><td port="frequency">timedelta</td></tr><tr><td>ranges</td><td port="ranges">List[TimeRangeModel]</td></tr><tr><td>leap_day_adjustment</td><td port="leap_day_adjustment">Optional[LeapDayAdjustmentType]</td></tr><tr><td>time_interval_type</td><td port="time_interval_type">TimeIntervalType</td></tr><tr><td>timezone</td><td port="timezone">TimeZone</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.DateTimeDimensionModel&#xA;&#xA;Defines a time dimension where timestamps translate to datetime objects.&#\
xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dimensions:e -> "dsgrid.config.dimensions.DateTimeDimensionModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.DimensionModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>DimensionModel</b></td></tr><tr><td>name</td><td port="name">str</td></tr><tr><td>display_name</td><td port="display_name">str</td></tr><tr><td>dimension_query_name</td><td port="dimension_query_name">Optional[str]</td></tr><tr><td>dimension_type</td><td port="dimension_type">DimensionType</td></tr><tr><td>dimension_id</td><td port="dimension_id">Optional[str]</td></tr><tr><td>version</td><td port="version">Optional[str]</td></tr><tr><td>module</td><td port="module">Optional[str]</td></tr><tr><td>class_name</td><td port="class_name">str</td></tr><tr><td>cls</td><td port="cls">Optional[Any]</td></tr><tr><td>description</td><td port="description">str</td></tr><tr><td>id</td><td port="id">Optional[str]</td></tr><tr><td>key</td><td port="key">Optional[str]</td></tr><tr><td>rev</td><td port="rev">Optional[str]</td></tr><tr><td>filename</td><td port="filename">Optional[str]</td></tr><tr><td>file_hash</td><td port="file_hash">Optional[str]</td></tr><tr><td>records</td><td port="records">list[]</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.DimensionModel&#xA;&#xA;Defines a non-time dimension&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dimensions:e -> "dsgrid.config.dimensions.DimensionModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.DimensionReferenceModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>DimensionReferenceModel</b></td></tr><tr><td>dimension_type</td><td port="dimension_type">DimensionType</td></tr><tr><td>dimension_id</td><td port="dimension_id">str</td></tr><tr><td>version</td><td port="version">str</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.DimensionReferenceModel&#xA;&#xA;Reference to a dimension stored in the registry&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dimension_references:e -> "dsgrid.config.dimensions.DimensionReferenceModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.NoOpTimeDimensionModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>NoOpTimeDimensionModel</b></td></tr><tr><td>name</td><td port="name">str</td></tr><tr><td>display_name</td><td port="display_name">str</td></tr><tr><td>dimension_query_name</td><td port="dimension_query_name">Optional[str]</td></tr><tr><td>dimension_type</td><td port="dimension_type">DimensionType</td></tr><tr><td>dimension_id</td><td port="dimension_id">Optional[str]</td></tr><tr><td>version</td><td port="version">Optional[str]</td></tr><tr><td>module</td><td port="module">Optional[str]</td></tr><tr><td>class_name</td><td port="class_name">str</td></tr><tr><td>cls</td><td port="cls">Optional[Any]</td></tr><tr><td>description</td><td port="description">str</td></tr><tr><td>id</td><td port="id">Optional[str]</td></tr><tr><td>key</td><td port="key">Optional[str]</td></tr><tr><td>rev</td><td port="rev">Optional[str]</td></tr><tr><td>time_type</td><td port="time_type">TimeDimensionType</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.NoOpTimeDimensionModel&#xA;&#xA;Defines a NoOp time dimension.&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dimensions:e -> "dsgrid.config.dimensions.NoOpTimeDimensionModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.RepresentativePeriodTimeDimensionModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>RepresentativePeriodTimeDimensionModel</b></td></tr><tr><td>name</td><td port="name">str</td></tr><tr><td>display_name</td><td port="display_name">str</td></tr><tr><td>dimension_query_name</td><td port="dimension_query_name">Optional[str]</td></tr><tr><td>dimension_type</td><td port="dimension_type">DimensionType</td></tr><tr><td>dimension_id</td><td port="dimension_id">Optional[str]</td></tr><tr><td>version</td><td port="version">Optional[str]</td></tr><tr><td>module</td><td port="module">Optional[str]</td></tr><tr><td>class_name</td><td port="class_name">str</td></tr><tr><td>cls</td><td port="cls">Optional[Any]</td></tr><tr><td>description</td><td port="description">str</td></tr><tr><td>id</td><td port="id">Optional[str]</td></tr><tr><td>key</td><td port="key">Optional[str]</td></tr><tr><td>rev</td><td port="rev">Optional[str]</td></tr><tr><td>time_type</td><td port="time_type">TimeDimensionType</td></tr><tr><td>measurement_type</td><td port="measurement_type">MeasurementType</td></tr><tr><td>format</td><td port="format">RepresentativePeriodFormat</td></tr><tr><td>ranges</td><td port="ranges">List[MonthRangeModel]</td></tr><tr><td>time_interval_type</td><td port="time_interval_type">TimeIntervalType</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.RepresentativePeriodTimeDimensionModel&#xA;&#xA;Defines a representative time dimension.&#xA;"];
   "dsgrid.config.dataset_config.DatasetConfigModel":dimensions:e -> "dsgrid.config.dimensions.RepresentativePeriodTimeDimensionModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.dataset.models.PivotedTableFormatModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>PivotedTableFormatModel</b></td></tr><tr><td>format_type</td><td port="format_type">Literal[TableFormatType.PIVOTED]</td></tr><tr><td>pivoted_dimension_type</td><td port="pivoted_dimension_type">DimensionType</td></tr></table>>,
      tooltip="dsgrid.dataset.models.PivotedTableFormatModel&#xA;&#xA;Base data model for all dsgrid data models&#xA;"];
   "dsgrid.config.dataset_config.OneTableDataSchemaModel":table_format:e -> "dsgrid.dataset.models.PivotedTableFormatModel":_root:w   [arrowhead=noneteetee,
      arrowtail=nonenone];
   "dsgrid.dataset.models.UnpivotedTableFormatModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>UnpivotedTableFormatModel</b></td></tr><tr><td>format_type</td><td port="format_type">Literal[TableFormatType.UNPIVOTED]</td></tr><tr><td>value_column</td><td port="value_column">str</td></tr></table>>,
      tooltip="dsgrid.dataset.models.UnpivotedTableFormatModel&#xA;&#xA;Base data model for all dsgrid data models&#xA;"];
   "dsgrid.config.dataset_config.OneTableDataSchemaModel":table_format:e -> "dsgrid.dataset.models.UnpivotedTableFormatModel":_root:w   [arrowhead=noneteetee,
      arrowtail=nonenone];
   "dsgrid.config.dataset_config.StandardDataSchemaModel":table_format:e -> "dsgrid.dataset.models.PivotedTableFormatModel":_root:w   [arrowhead=noneteetee,
      arrowtail=nonenone];
   "dsgrid.config.dataset_config.StandardDataSchemaModel":table_format:e -> "dsgrid.dataset.models.UnpivotedTableFormatModel":_root:w   [arrowhead=noneteetee,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.TimeRangeModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>TimeRangeModel</b></td></tr><tr><td>start</td><td port="start">str</td></tr><tr><td>end</td><td port="end">str</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.TimeRangeModel&#xA;&#xA;Defines a continuous range of time.&#xA;"];
   "dsgrid.config.dimensions.AnnualTimeDimensionModel":ranges:e -> "dsgrid.config.dimensions.TimeRangeModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.DateTimeDimensionModel":ranges:e -> "dsgrid.config.dimensions.TimeRangeModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
   "dsgrid.config.dimensions.MonthRangeModel"   [label=<<table border="0" cellborder="1" cellspacing="0"><tr><td port="_root" colspan="2"><b>MonthRangeModel</b></td></tr><tr><td>start</td><td port="start">int</td></tr><tr><td>end</td><td port="end">int</td></tr></table>>,
      tooltip="dsgrid.config.dimensions.MonthRangeModel&#xA;&#xA;Defines a continuous range of time.&#xA;"];
   "dsgrid.config.dimensions.RepresentativePeriodTimeDimensionModel":ranges:e -> "dsgrid.config.dimensions.MonthRangeModel":_root:w   [arrowhead=crownone,
      arrowtail=nonenone];
}
Fields:
Validators:
field data_classification: DataClassificationType [Required]

Data security classification (e.g., low, moderate, high)

Validated by:
field data_schema: StandardDataSchemaModel | OneTableDataSchemaModel [Required]

Schema (table layouts) used for writing out the dataset

Validated by:
field data_source: str [Required]

Data source name, e.g. ‘ComStock’.

Validated by:
field dataset_id: str [Required]

Unique dataset identifier.

Validated by:
field dataset_qualifier_metadata: QuantityModel | GrowthRateModel | None = QuantityModel(dataset_qualifier_type=<DatasetQualifierType.QUANTITY: 'quantity'>)

Additional metadata to include related to the dataset_qualifier

Validated by:
field dataset_type: InputDatasetType [Required]

Input dataset type.

Validated by:
field dataset_version: str | None = None

The version of the dataset.

Validated by:
field description: str [Required]

A detailed description of the dataset.

Validated by:
field dimension_references: List[DimensionReferenceModel] = []

List of registered dimension references that make up the dimensions of dataset.

Validated by:
field dimensions: List[DimensionModel | DateTimeDimensionModel | AnnualTimeDimensionModel | RepresentativePeriodTimeDimensionModel | NoOpTimeDimensionModel] = []

List of dimensions that make up the dimensions of dataset. They will be automatically registered during dataset registration and then converted to dimension_references.

Constraints:
  • func = <function handle_dimension_union at 0x7ff05b0d9ea0>

Validated by:
field enable_unit_conversion: bool = True

If the dataset uses its dimension mapping for the metric dimension to also perform unit conversion, then this value should be false.

Validated by:
field id: str | None = None (alias '_id')

Registry database ID

Validated by:
field key: str | None = None (alias '_key')

Registry database key

Validated by:
field origin_contributors: List[str] = []

List of origin data contributor’s first and last names e.g., [“Harry Potter”, “Ronald Weasley”]

Validated by:
field origin_creator: str [Required]

Origin data creator’s name (first and last)

Validated by:
field origin_date: str [Required]

Date the source data was generated

Validated by:
field origin_organization: str [Required]

Origin organization name, e.g., NREL

Validated by:
field origin_project: str [Required]

Origin project name

Validated by:
field origin_version: str [Required]

Version of the origin data

Validated by:
field rev: str | None = None (alias '_rev')

Registry database revision

Validated by:
field sector_description: str | None = None

Sectoral description (e.g., residential, commercial, industrial, transportation, electricity)

Validated by:
field source: str [Required]

Source of the data (text description or link)

Validated by:
field tags: List[str] = []

List of data tags

Validated by:
field trivial_dimensions: List[DimensionType] | None = []

List of trivial dimensions (i.e., 1-element dimensions) that do not exist in the load_data_lookup. List the dimensions by dimension type.

Validated by:
field use_project_geography_time_zone: bool = False

If true, time zones will be applied from the project’s geography dimension. If false, the dataset’s geography dimension records must provide a time zone column.

Validated by:
field user_defined_metadata: Dict = {}

Additional user defined metadata fields

Validated by:
field version: str | None = None

Version, generated by dsgrid

Validated by:
validator check_dataset_id  »  dataset_id[source]

Check dataset ID validity

validator check_files  »  dimensions[source]

Validate dimension files are unique across all dimensions

validator check_names  »  dimensions[source]

Validate dimension names are unique across all dimensions.

validator check_time_not_trivial  »  trivial_dimensions[source]
validator check_time_zone  »  all fields[source]

Validate whether required time zone information is present.

validator handle_legacy_fields  »  all fields[source]