Dataset Query Concepts

A dataset query remaps a registered dataset’s dimensions without involving a project. This is useful when you want to map a dataset to alternate dimensions (e.g., aggregate county-level data to state-level) as a standalone operation.

If you need to combine multiple datasets, apply result-level aggregations, filters, or reports, use a project query instead. See Project Query Concepts.

How It Works

A dataset query has three phases:

  1. Resolve mappings — dsgrid looks up registered dimension mappings between the dataset’s current dimensions and the target dimensions you specify. You do not supply mapping IDs directly — dsgrid finds them automatically using a graph traversal of the dimension mapping registry.

  2. Apply mappings — dsgrid applies each mapping to transform the data.

  3. Write results — The mapped data is written to the output directory in the specified format (Parquet or CSV).

The DatasetQueryModel

A dataset query is defined by a JSON5 file with these fields:

{
  // Required
  name: "my-aggregated-dataset",
  dataset_id: "my-registered-dataset-id",
  to_dimension_references: [
    {
      type: "geography",
      dimension_id: "<target-dimension-uuid>",
      version: "1.0.0",
    },
  ],

  // Optional
  mapping_plan: null,  // leave null; see note below
  wrap_time_allowed: false,
  time_based_data_adjustment: {},

  // Result params (limited subset)
  result: {
    table_format: {format_type: "stacked"},
    output_format: "parquet",
    sort_columns: [],
    single_output_file: false,
  },
}

to_dimension_references

This is the key field. Each entry identifies a target dimension in the registry by its type, dimension_id, and version. dsgrid will find the registered mapping between the dataset’s current dimension and the specified target.

  • At least one reference is required.

  • You cannot specify two references of the same dimension type.

  • If the target dimension is identical to the dataset’s current dimension (same ID and version), dsgrid skips it — no mapping needed.

mapping_plan

User-defined mapping plans (and the checkpoint files that depend on them) are not yet supported for dataset queries. Leave mapping_plan as null; dsgrid builds a default plan from the resolved mappings. Mapping plans are supported in project queries; see the Map a Dataset tutorial.

wrap_time_allowed

When true, allows the dataset’s time to be wrapped to the target time dimension if the two differ.

Supported Result Parameters

Dataset queries support a limited subset of the result parameters available in project queries:

Parameter

Supported

Notes

table_format

Yes

Stacked or pivoted

output_format

Yes

"parquet" or "csv"

sort_columns

Yes

Sort result by these columns. On the Spark backend, the output file is only sorted if single_output_file is also true.

single_output_file

Yes

Write one file instead of a directory of part files. Only matters on Spark; DuckDB always writes a single file.

aggregations

No

Aggregation happens through mapping types

dimension_filters

No

Filter before querying or post-process results

reports

No

Use project queries for reports

replace_ids_with_names

No

Post-process results if needed

time_zone

No

Not available for dataset queries

Important

Aggregation in a dataset query is achieved through the dimension mapping type (e.g., many_to_one_aggregation), not through a result-level aggregation parameter. This is the key difference from project queries, which support explicit aggregations in the result block.

Mapping Resolution

When you specify a to_dimension_references entry, dsgrid resolves the required mapping automatically:

  1. It looks up the dataset’s current dimension for that dimension type.

  2. It searches the dimension mapping registry graph for the shortest chain of registered mappings from the dataset’s dimension to the target dimension. This is usually a single mapping, but dsgrid can chain several (e.g., county → state → census region). It uses the latest version of each mapping.

  3. If a path exists, the mappings are applied in order. If not, the query fails.

This means the dimension mappings must already be registered before you run the query. See How to Create Dimension Mappings. Time is the exception: time conversion does not use registered mappings, so a time reference only needs to identify the target time dimension.

Comparison with Project Queries

Feature

Dataset Query

Project Query

Command

dsgrid query dataset

dsgrid query project

Requires project

No

Yes

Multiple datasets

No (single dataset)

Yes (combine with expressions)

Mapping target

Any registered dimension

Project base dimensions

Aggregation method

Through mapping types

Through result.aggregations or mapping types

Filters

Not supported

Supported (pre and post)

Reports

Not supported

Supported

Caching

Output only

Intermediate + output caching

CLI Commands

Generate a query template

dsgrid query dataset create-query <name> <dataset-id>

This writes a dataset_query.json5 file with empty to_dimension_references that you populate with the target dimensions.

Run a query

dsgrid query dataset run dataset_query.json5 -o output/

Options:

  • -o, --output — Output directory (default: query_output)

  • --overwrite — Overwrite existing results

Next Steps