Core Concepts

This section explains the key concepts and architecture behind STRIDE’s data pipeline.

Overview

STRIDE transforms raw energy data into hourly electricity demand projections through a multi-stage pipeline:

1. Data Download - Retrieve datasets from remote repositories (e.g., GitHub releases)

2. Data Validation - Register and validate data using dsgrid’s dimension mapping system

3. Computation - Calculate energy projections using dbt (data build tool) SQL transformations

Each stage is designed to be modular and customizable, allowing users to swap out datasets, adjust validation rules, or modify calculation logic.

• Data Download
  • Default Data Repository
  • How Downloads Work
  • CLI Commands
  • Storage Location
  • Using Test Data
  • Error Handling
• Data Validation with dsgrid
  • What is dsgrid?
  • The Validation Process
  • What Gets Validated
  • Validation Errors
  • Scenarios and Alternative Datasets
  • Related Topics
• Customizing Dataset Checks
  • Available Checks
  • Using Environment Variables
  • Programmatic Configuration
  • When to Customize Checks
  • Check Behavior
  • Performance Considerations
  • Related Topics
• dbt Computation
  • Why dbt?
  • Project Structure
  • How dbt is Invoked
  • Data Flow
  • The Override Mechanism
  • Debugging dbt
  • Customizing Calculations
  • Related Topics