Installation

Python Environment

dsgrid requires python=3.10 or later. If you do not already have a python environment with python>=3.10, we recommend using Conda to help manage your python packages and environments.

Steps to make a dsgrid Conda environment:

1. Download and install Conda if it is not already installed. We recommend Miniconda over Anaconda because it has a smaller installation size.

2. Create a suitable environment.

$ conda create -n dsgrid python=3.10

3. Activate the environment:

$ conda activate dsgrid

dsgrid’s key dependencies are an ArangoDB registry, which can be shared or standalone, and Apache Spark. Apache Spark requires Java, so check if you have it. Both of these commands must work:

Mac/LinuxWindows

$ java --version
openjdk 11.0.12 2021-07-20
$ echo $JAVA_HOME
/Users/dthom/brew/Cellar/openjdk@11/11.0.12

$ # If you don't have java installed:
$ conda install openjdk

Package Installation

With ssh keys:

pip install git+ssh://git@github.com/dsgrid/dsgrid.git@main

Or from http:

pip install git+https://github.com/dsgrid/dsgrid.git@main

Todo

pipy.org/pip installation not available yet.

Registry

NREL Shared Registry

The current dsgrid registry is hosted on a VM in the NREL data center. The database URL is http://dsgrid-registry.hpc.nrel.gov:8529. All configuration information is stored in the database and all dataset files are stored on the the NREL HPC.

You can list projects, datasets, and dimensions with the dsgrid CLI tool from any system within NREL. To query the data you must be on the NREL HPC.

Todo

Steps to replicate the database to another system

Standalone Registry

To use dsgrid in your own computational environment, you will need to run ArangoDB and either set up a new dsgrid registry or load a dsgrid registry that has been written to disk. Please see the how-to guide on setting up a standalone dsgrid registry.

Apache Spark

• NREL High Performance Computing: How to Start a Spark Cluster on Kestrel

• Standalone resources: [TODO: Provide link]

Test your installation

If you’re running dsgrid at NREL and using the shared registry, you can test your installation with this command:

$ dsgrid -u http://dsgrid-registry.hpc.nrel.gov:8529 -N standard-scenarios registry projects list

You can test your installation similarly if you are using a different registry, just change the ArangoDB URL (-u) and database name (-N) arguments to match your set-up.

Save your configuration

Running dsgrid config create stores key information for working with dsgrid in a config file at ~/.dsgrid.json5. Currently, dsgrid only supports offline mode, and the other key information to store is the ArangoDB URL and the name of the dsgrid registry. The parameters in the config file are the default values used by the command-line interface.

The appropriate configuration for using the shared registry at NREL is:

$ dsgrid config create -u http://dsgrid-registry.hpc.nrel.gov:8529 -N standard-scenarios --offline

Similar to testing your installation, you can save the correct configurations for other set-ups by changing the ArangoDB URL (-u) and database name (-N) arguments of the above command.

Todo

Access from AWS