CLI Reference¶

dsgrid¶

dsgrid commands

dsgrid [OPTIONS] COMMAND [ARGS]...

Options

-c, --console-level <console_level>¶

Console log level.

Default:: info

-f, --file-level <file_level>¶

File log level.

Default:: info

-l, --log-file <log_file>¶: Log to this file.

-n, --no-prompts¶

Do not prompt.

Default:: False

--offline, --online¶

Run registry commands in offline mode. WARNING: any commands you perform in offline mode run the risk of being out-of-sync with the latest dsgrid registry, and any write commands will not be officially synced with the remote registry

Default:: False

--timings, --no-timings¶

Enable tracking of function timings.

Default:: False

-N, --database-name <database_name>¶: Database name

-u, --url <url>¶: Database URL. Ex: http://localhost:8529

-U, --username <username>¶: Database username

-P, --password <password>¶: dsgrid registry password. Will prompt unless it is passed or the username matches the runtime config file.

-r, --reraise-exceptions¶

Re-raise any dsgrid exception. Default is to log the exception and exit.

Default:: False

-s, --scratch-dir <scratch_dir>¶: Base directory for dsgrid temporary directories. Must be accessible on all compute nodes. Defaults to the current directory.

Environment variables

DSGRID_REGISTRY_DATABASE_NAME: Provide a default for -N

DSGRID_REGISTRY_DATABASE_URL: Provide a default for -u

config¶

Config commands

dsgrid config [OPTIONS] COMMAND [ARGS]...

create¶

Create a local dsgrid runtime configuration file.

dsgrid config create [OPTIONS]

Options

--timings, --no-timings¶

Enable tracking of function timings.

Default:: False

-N, --database-name <database_name>¶: Database name

-u, --url <url>¶: Database URL. Ex: http://localhost:8529

-U, --username <username>¶: Database username

-P, --password <password>¶: Database username

-o, --offline¶

Default:: False

--console-level <console_level>¶

Console log level.

Default:: info

--file-level <file_level>¶

File log level.

Default:: info

-r, --reraise-exceptions¶

Re-raise any dsgrid exception. Default is to log the exception and exit.

Default:: False

-s, --scratch-dir <scratch_dir>¶: Base directory for dsgrid temporary directories. Must be accessible on all compute nodes. Defaults to the current directory.

download¶

Download a dataset.

dsgrid download [OPTIONS] DATASET

Arguments

DATASET¶: Required argument

install-notebooks¶

Install dsgrid notebooks to a local path.

dsgrid install-notebooks [OPTIONS]

Options

-p, --path <path>¶

Path to install dsgrid notebooks.

Default:: /home/runner

-f, --force¶

If true, overwrite existing files.

Default:: False

query¶

Query group commands

dsgrid query [OPTIONS] COMMAND [ARGS]...

composite-dataset¶

Composite dataset group commands

dsgrid query composite-dataset [OPTIONS] COMMAND [ARGS]...

create_dataset¶

Run a query to create a composite dataset.

dsgrid query composite-dataset create_dataset [OPTIONS] QUERY_DEFINITION_FILE

Options

-o, --output <output>¶

Output directory for query results

Default:: query_output

--load-cached-table, --no-load-cached-table¶

Try to load a cached table if one exists.

Default:: True

--force¶

Overwrite results directory if it exists.

Default:: False

Arguments

QUERY_DEFINITION_FILE¶: Required argument

run¶

Run a query on a composite dataset.

dsgrid query composite-dataset run [OPTIONS] QUERY_DEFINITION_FILE

Options

-o, --output <output>¶

Output directory for query results

Default:: query_output

--load-cached-table, --no-load-cached-table¶

Try to load a cached table if one exists.

Default:: True

--force¶

Overwrite results directory if it exists.

Default:: False

Arguments

QUERY_DEFINITION_FILE¶: Required argument

project¶

Project group commands

dsgrid query project [OPTIONS] COMMAND [ARGS]...

create¶

Create a default query file for a dsgrid project.

dsgrid query project create [OPTIONS] QUERY_NAME PROJECT_ID DATASET_ID

Options

-F, --filters <filters>¶

Add a dimension filter. Requires user customization.

Options:: expression | expression_raw | column_operator | between_column_operator | subset | supplemental_column_operator

-a, --aggregation-function <aggregation_function>¶

Aggregation function for any included default aggregations.

Default:: sum

-d, --default-per-dataset-aggregation¶

Add default per-dataset aggregration.

Default:: False

-f, --query-file <query_file>¶

Query file to create.

Default:: query.json5

-r, --default-result-aggregation¶

Add default result aggregration.

Default:: False

--force¶

Overwrite query file if it exists.

Default:: False

--remote-path <remote_path>¶

Path to dsgrid remote registry

Default:: s3://nrel-dsgrid-registry

Arguments

QUERY_NAME¶: Required argument

PROJECT_ID¶: Required argument

DATASET_ID¶: Required argument

create-derived-dataset-config¶

Create a derived dataset configuration and dimensions from a query result.

dsgrid query project create-derived-dataset-config [OPTIONS] SRC DST

Options

--remote-path <remote_path>¶

Path to dsgrid remote registry

Default:: s3://nrel-dsgrid-registry

--force¶

Overwrite results directory if it exists.

Default:: False

Arguments

SRC¶: Required argument

DST¶: Required argument

run¶

Run a query on a dsgrid project.

dsgrid query project run [OPTIONS] QUERY_DEFINITION_FILE

Options

--persist-intermediate-table, --no-persist-intermediate-table¶

Persist the intermediate table to the filesystem to allow for reuse.

Default:: True

-z, --zip-file¶

Create a zip file containing all output files.

Default:: False

--remote-path <remote_path>¶

Path to dsgrid remote registry

Default:: s3://nrel-dsgrid-registry

-o, --output <output>¶

Output directory for query results

Default:: query_output

--load-cached-table, --no-load-cached-table¶

Try to load a cached table if one exists.

Default:: True

--force¶

Overwrite results directory if it exists.

Default:: False

Arguments

QUERY_DEFINITION_FILE¶: Required argument

validate¶

dsgrid query project validate [OPTIONS] QUERY_FILE

Arguments

QUERY_FILE¶: Required argument

registry¶

Manage a registry.

dsgrid registry [OPTIONS] COMMAND [ARGS]...

Options

--remote-path <remote_path>¶

path to dsgrid remote registry

Default:: s3://nrel-dsgrid-registry

data-sync¶

Sync the official dsgrid registry data to the local system.

dsgrid registry data-sync [OPTIONS]

Options

-P, --project-id <project_id>¶: Sync latest dataset(s) version based on Project ID

-D, --dataset-id <dataset_id>¶: Sync latest dataset version based on Dataset ID

datasets¶

Dataset subcommands

dsgrid registry datasets [OPTIONS] COMMAND [ARGS]...

dump¶

Dump a dataset config file from the registry.

dsgrid registry datasets dump [OPTIONS] DATASET_ID

Options

-v, --version <version>¶: Version to dump; defaults to latest

-d, --directory <directory>¶: Directory in which to create the config file

--force¶

Overwrite files if they exist.

Default:: False

Arguments

DATASET_ID¶: Required argument

list¶

List the registered dimensions.

dsgrid registry datasets list [OPTIONS]

Options

-f, --filter <filter>¶

filter table with a case-insensitive expression in the format ‘field operation value’, accept multiple flags

valid operations: [‘==’, ‘!=’, ‘contains’, ‘not contains’]

example:

-f ‘Submitter == username’ -f ‘Description contains sector’

register¶

dsgrid registry datasets register [OPTIONS] DATASET_CONFIG_FILE DATASET_PATH

Options

-l, --log-message <log_message>¶: Required reason for submission

Arguments

DATASET_CONFIG_FILE¶: Required argument

DATASET_PATH¶: Required argument

update¶

Update an existing dataset registry.

dsgrid registry datasets update [OPTIONS] DATASET_CONFIG_FILE

Options

-d, --dataset-id <dataset_id>¶: Required dataset ID

-l, --log-message <log_message>¶: Required reason for submission

-t, --update-type <update_type>¶

Required

Options:: major | minor | patch

-v, --version <version>¶: Required Version to update; must be the current version.

Arguments

DATASET_CONFIG_FILE¶: Required argument

dimension-mappings¶

Dimension mapping subcommands

dsgrid registry dimension-mappings [OPTIONS] COMMAND [ARGS]...

dump¶

Dump a dimension mapping config file (and any related data) from the registry.

dsgrid registry dimension-mappings dump [OPTIONS] DIMENSION_MAPPING_ID

Options

-v, --version <version>¶: Version to dump; defaults to latest

-d, --directory <directory>¶: Directory in which to create config and data files

--force¶

Overwrite files if they exist.

Default:: False

Arguments

DIMENSION_MAPPING_ID¶: Required argument

list¶

List the registered dimension mappings.

dsgrid registry dimension-mappings list [OPTIONS]

Options

-f, --filter <filter>¶

filter table with a case-insensitive expression in the format ‘field operation value’, accept multiple flags

valid operations: [‘==’, ‘!=’, ‘contains’, ‘not contains’]

example:

-f ‘Submitter == username’ -f ‘Description contains sector’

register¶

dsgrid registry dimension-mappings register [OPTIONS]
                                            DIMENSION_MAPPING_CONFIG_FILE

Options

-l, --log-message <log_message>¶: Required reason for submission

Arguments

DIMENSION_MAPPING_CONFIG_FILE¶: Required argument

update¶

Update an existing dimension mapping registry.

dsgrid registry dimension-mappings update [OPTIONS]
                                          DIMENSION_MAPPING_CONFIG_FILE

Options

-d, --dimension-mapping-id <dimension_mapping_id>¶: Required dimension mapping ID

-l, --log-message <log_message>¶: Required reason for submission

-t, --update-type <update_type>¶

Required

Options:: major | minor | patch

-v, --version <version>¶: Required Version to update; must be the current version.

Arguments

DIMENSION_MAPPING_CONFIG_FILE¶: Required argument

dimensions¶

Dimension subcommands

dsgrid registry dimensions [OPTIONS] COMMAND [ARGS]...

dump¶

Dump a dimension config file (and any related data) from the registry.

dsgrid registry dimensions dump [OPTIONS] DIMENSION_ID

Options

-v, --version <version>¶: Version to dump; defaults to latest

-d, --directory <directory>¶: Directory in which to create config and data files

--force¶

Overwrite files if they exist.

Default:: False

Arguments

DIMENSION_ID¶: Required argument

list¶

List the registered dimensions.

dsgrid registry dimensions list [OPTIONS]

Options

-f, --filter <filter>¶

filter table with a case-insensitive expression in the format ‘field operation value’, accept multiple flags

valid operations: [‘==’, ‘!=’, ‘contains’, ‘not contains’]

example:

-f ‘Submitter == username’ -f ‘Description contains sector’

register¶

dsgrid registry dimensions register [OPTIONS] DIMENSION_CONFIG_FILE

Options

-l, --log-message <log_message>¶: Required reason for submission

Arguments

DIMENSION_CONFIG_FILE¶: Required argument

update¶

Update an existing dimension registry.

dsgrid registry dimensions update [OPTIONS] DIMENSION_CONFIG_FILE

Options

-d, --dimension-id <dimension_id>¶: Required dimension ID

-l, --log-message <log_message>¶: Required reason for submission

-t, --update-type <update_type>¶

Required

Options:: major | minor | patch

-v, --version <version>¶: Required Version to update; must be the current version.

Arguments

DIMENSION_CONFIG_FILE¶: Required argument

list¶

List the contents of a registry.

dsgrid registry list [OPTIONS]

projects¶

Project subcommands

dsgrid registry projects [OPTIONS] COMMAND [ARGS]...

dump¶

Dump a project config file from the registry.

dsgrid registry projects dump [OPTIONS] PROJECT_ID

Options

-v, --version <version>¶: Version to dump; defaults to latest

-d, --directory <directory>¶: Directory in which to create the config file

--force¶

Overwrite files if they exist.

Default:: False

Arguments

PROJECT_ID¶: Required argument

list¶

List the registered projects.

dsgrid registry projects list [OPTIONS]

Options

-f, --filter <filter>¶

filter table with a case-insensitive expression in the format ‘field operation value’, accept multiple flags

valid operations: [‘==’, ‘!=’, ‘contains’, ‘not contains’]

example:

-f ‘Submitter == username’ -f ‘Description contains sector’

list-dimension-query-names¶

List the project’s dimension query names.

dsgrid registry projects list-dimension-query-names [OPTIONS] PROJECT_ID

Options

-b, --exclude-base¶

Exclude base dimension query names.

Default:: False

-S, --exclude-subset¶

Exclude subset dimension query names.

Default:: False

-s, --exclude-supplemental¶

Exclude supplemental dimension query names.

Default:: False

Arguments

PROJECT_ID¶: Required argument

register¶

dsgrid registry projects register [OPTIONS] PROJECT_CONFIG_FILE

Options

-l, --log-message <log_message>¶: Required reason for submission

Arguments

PROJECT_CONFIG_FILE¶: Required argument

register-and-submit-dataset¶

dsgrid registry projects register-and-submit-dataset [OPTIONS]

Options

-c, --dataset-config-file <dataset_config_file>¶: Required Dataset config file

-d, --dataset-path <dataset_path>¶: Required

-m, --dimension-mapping-file <dimension_mapping_file>¶: File containing dimension mappings to register with the dataset

-r, --dimension-mapping-references-file <dimension_mapping_references_file>¶: dimension mapping references file. Mutually exclusive with dimension_mapping_file. Use it when the mappings are already registered.

-a, --autogen-reverse-supplemental-mappings <autogen_reverse_supplemental_mappings>¶

For any dimension listed here, if the dataset’s dimension is a project’s supplemental dimension and no mapping is provided, create a reverse mapping from that supplemental dimension.

Options:: metric | geography | sector | subsector | time | weather_year | model_year | scenario

-p, --project-id <project_id>¶: Required project identifier

-l, --log-message <log_message>¶: Required reason for submission

submit-dataset¶

Submit a dataset to a dsgrid project.

dsgrid registry projects submit-dataset [OPTIONS]

Options

-d, --dataset-id <dataset_id>¶: Required dataset identifier

-p, --project-id <project_id>¶: Required project identifier

-m, --dimension-mapping-file <dimension_mapping_file>¶: dimension mapping file

-r, --dimension-mapping-references-file <dimension_mapping_references_file>¶: dimension mapping references file. Mutually exclusive with dimension_mapping_file. Use it when the mappings are already registered.

-a, --autogen-reverse-supplemental-mappings <autogen_reverse_supplemental_mappings>¶

For any dimension listed here, if the dataset’s dimension is a project’s supplemental dimension and no mapping is provided, create a reverse mapping from that supplemental dimension.

Options:: metric | geography | sector | subsector | time | weather_year | model_year | scenario

-l, --log-message <log_message>¶: Required reason for submission

update¶

Update an existing project registry.

dsgrid registry projects update [OPTIONS] PROJECT_CONFIG_FILE

Options

-p, --project-id <project_id>¶: Required project ID

-l, --log-message <log_message>¶: Required reason for submission

-t, --update-type <update_type>¶

Required

Options:: major | minor | patch

-v, --version <version>¶: Required Version to update; must be the current version.

Arguments

PROJECT_CONFIG_FILE¶: Required argument