Skip to content

Data

Setup-time facade for parquet-backed time series. For the architecture-level role, see Architecture → Data pipeline. Nothing in powerzoojax.data enters the compiled rollout.

Prefer load_jax_profiles(...) for env construction and load_signals(...) for inspection. load_data(...) covers raw-column layouts and direct table access; benchmark code standardizes on load_jax_profiles(...) and the parquet facade documented in Architecture → Data pipeline.

Semantic loading

from powerzoojax.data import DataLoader, signals as S

loader = DataLoader()
profiles = loader.load_jax_profiles(
    [S.LOAD_ACTUAL_MW, S.SOLAR_AVAILABLE_MW],
    source="gb",
    start_date="2025-04-01",
    end_date="2025-12-31",
    resample="30min",
)

Stable signal names

Power-system signals:

  • LOAD_ACTUAL_MW
  • LOAD_FORECAST_DA_MW
  • LOAD_FORECAST_P10_MW
  • LOAD_FORECAST_P50_MW
  • LOAD_FORECAST_P90_MW
  • SOLAR_AVAILABLE_MW
  • WIND_AVAILABLE_MW

Market index signals:

  • MARKET_MID_PRICE_APX
  • MARKET_MID_PRICE_N2EX
  • MARKET_MID_VOLUME_APX
  • MARKET_MID_VOLUME_N2EX

Data-center signals:

  • DC_CPU_UTIL
  • DC_MEM_UTIL
  • DC_NET_IN
  • DC_NET_OUT
  • DC_DISK_IO
  • DC_POWER_MW
  • DC_GPU_UTIL
  • DC_GPU_MEM_UTIL

Loader, manifest, registry

powerzoojax.data.data_loader.DataLoader(data_dir=None, manifest_dir=None)

Benchmark data facade backed by parquet files and manifest metadata.

Features: - JAX array output via :meth:load_jax_profiles (float32, GPU-ready) - Semantic signal loading via :meth:load_signals - Calendar / profile time-alignment across heterogeneous sources - Forecast-panel loading with issue_time / target_time - Legacy column-based loading via :meth:load_data - Resampling, date filtering, multi-dataset merge

Initialize DataLoader.

Parameters:

Name Type Description Default
data_dir Optional[Union[str, Path]]

Directory containing parquet files. If None, defaults to powerzoojax/data/parquet.

None
manifest_dir Optional[Union[str, Path]]

Directory containing manifest JSON files. If None, defaults to powerzoojax/data/manifests.

None

registry property

Expose the dataset registry for advanced querying.

load_jax_profiles(signals, *, source=None, region=None, start_date=None, end_date=None, resample=None, time_alignment=None, interpolation='linear', dtype=jnp.float32)

Load semantic signals and return as a JAX array.

This is the primary entry-point for env construction. It calls :meth:load_signals internally, then converts numeric columns to a jnp.ndarray of shape (T, n_signals).

The result is a JAX array on the default device (GPU when available), ready to be passed to make_trans_params(case, load_profiles=...).

Parameters:

Name Type Description Default
signals List[str]

Semantic signal names (e.g. ["load.actual_mw", "solar.available_mw"]).

required
source Optional[str]

Restrict lookup to a specific data source.

None
region Optional[str]

Filter by region (e.g. "NSW1").

None
start_date Optional[Union[str, datetime, date]]

Simulation time window start (inclusive).

None
end_date Optional[Union[str, datetime, date]]

Simulation time window end (inclusive).

None
resample Optional[str]

Target frequency ("5min", "30min", ...).

None
time_alignment Optional[Dict[str, str]]

Per-signal calendar-shift overrides.

None
interpolation str

Interpolation method when resampling.

'linear'
dtype dtype

JAX dtype for the output array. Default jnp.float32.

float32

Returns:

Type Description
ndarray

jnp.ndarray of shape (T, n_signals) with columns

ndarray

ordered as signals.

load_jax_array(df, columns=None, dtype=jnp.float32)

Convert a DataFrame (e.g. from :meth:load_signals) to JAX.

Parameters:

Name Type Description Default
df DataFrame

Source DataFrame.

required
columns Optional[List[str]]

Columns to extract (order preserved). If None, all numeric columns are used.

None
dtype dtype

Target JAX dtype.

float32

Returns:

Type Description
ndarray

jnp.ndarray of shape (n_rows, n_columns).

load_signals(signals, *, source=None, region=None, start_date=None, end_date=None, resample=None, time_alignment=None, interpolation='linear')

Load one or more semantic signals into a single DataFrame.

Parameters:

Name Type Description Default
signals List[str]

Semantic signal names (e.g. ["load.actual_mw", "solar.available_mw"]).

required
source Optional[str]

Restrict lookup to a specific data source ("gb", "aemo", "alibaba", ...).

None
region Optional[str]

Filter by region (e.g. "NSW1").

None
start_date Optional[Union[str, datetime, date]]

Simulation time window start (inclusive).

None
end_date Optional[Union[str, datetime, date]]

Simulation time window end (inclusive).

None
resample Optional[str]

Target frequency ("5min", "30min", ...).

None
time_alignment Optional[Dict[str, str]]

Per-signal calendar-shift overrides.

None
interpolation str

Interpolation method when resampling.

'linear'

Returns:

Type Description
DataFrame

DataFrame indexed by datetime (simulation timeline).

load_actual_series(signals, **kwargs)

Convenience wrapper: load only actual_series data.

load_forecast_panel(signals, *, source=None, region=None, start_date=None, end_date=None)

Load forecast-panel data with issue_time + target_time.

list_available_datasets()

List all available datasets (parquet files).

get_metadata(dataset_name)

Get metadata for a dataset.

get_available_columns(dataset_name)

Get available columns for a dataset.

get_date_range(dataset_name)

Get date range for a dataset.

find_datasets_for_columns(columns)

Find which datasets contain the requested columns.

load_data(dataset_name=None, columns=None, start_date=None, end_date=None, validate_dates=True, merge_how='inner', resample=None, interpolation='linear')

Load data from parquet file(s) with optional resampling.

If dataset_name is not provided, automatically finds datasets based on columns.

print_dataset_info(dataset_name)

Print information about a dataset.

get_summary()

Get summary of currently loaded data.

powerzoojax.data.manifest.DatasetManifest(name, source, data_type, time_mode, resolution, parquet_file, column_map=dict(), index_map=dict(), derived=dict(), normalize=dict(), data_epoch=None, cyclical=False, region_values=list(), date_range=None, metadata_json=None, source_url=None, source_urls=None, source_organization=None) dataclass

Declarative description of one dataset.

A manifest is the single source of truth that tells the data layer: * which parquet file to read, * how to map raw columns to semantic signals, * how to handle time alignment (calendar vs. profile), and * whether data can be tiled cyclically.

signals property

All semantic signals this manifest can provide (mapped + derived).

raw_columns_needed property

Raw columns that must be read from the parquet file.

from_json(path) classmethod

Load a manifest from a JSON file.

to_dict()

Serialize to a plain dict (JSON-friendly).

powerzoojax.data.registry.DatasetRegistry(manifest_dir=None)

Load all manifest JSONs and build a reverse index from signal -> datasets.

find_by_signal(signal, source=None, data_type=None)

Return manifests that provide signal, optionally filtered.

resolve_signals(signals, source=None)

Map each requested signal to the best matching manifest.

Returns {signal: manifest}. Raises if any signal cannot be resolved.

powerzoojax.data.alignment.TimeAligner

Map raw-data timestamps onto a unified simulation timeline.

align_calendar(df, sim_start, sim_end, align_from=None, time_col='datetime') staticmethod

Shift a calendar-mode dataset onto [sim_start, sim_end].

align_profile(df, sim_start, sim_end, resolution, time_col='datetime') staticmethod

Tile a profile cyclically to cover [sim_start, sim_end].

Uses jnp.tile so the tiled numeric data stays on the JAX default device (GPU when available).

tile_profile_jax(values, n_needed) staticmethod

Tile a profile array to length n_needed using jnp.tile.

Parameters

values : (n_profile, n_signals) JAX array. n_needed : target number of time steps.

Returns

(n_needed, n_signals) JAX array, dtype float32.

align(df, *, time_mode, sim_start, sim_end, align_from=None, resolution='30min', time_col='datetime') classmethod

Dispatch to the correct alignment strategy.

Frozen splits

from powerzoojax.data import (
    GB_TRAIN_START, GB_TRAIN_END,
    GB_IID_START, GB_IID_END,
    AUSGRID_TRAIN_START, AUSGRID_TRAIN_END,
    AUSGRID_IID_START, AUSGRID_IID_END,
    AUSGRID_SUMMER_START, AUSGRID_SUMMER_END,
    gb_windows,
    ausgrid_windows,
)

powerzoojax.data.splits.gb_windows()

Return ((train_start, train_end), (iid_start, iid_end)).

powerzoojax.data.splits.ausgrid_windows()

Return (train_window, iid_window, summer_ood_window).

Ausgrid utilities

from powerzoojax.data import (
    AUSGRID_FEEDER_POOLS,
    AUSGRID_TRAIN_POOL,
    AUSGRID_ZONE_HOLDOUT_POOL,
    get_ausgrid_split,
    get_feeder_substations,
    select_full_coverage_substations,
)

powerzoojax.data.ausgrid_utils.get_ausgrid_split(role)

Return (start, end, substation_list) for a named split role.

Parameters:

Name Type Description Default
role str

One of "train", "iid", "summer_ood", "zone_holdout".

required

Returns:

Type Description
Tuple[str, str, List[str]]

Tuple of (ISO-date start, ISO-date end, list of substation names).

Raises:

Type Description
KeyError

if role is not recognised.

powerzoojax.data.ausgrid_utils.get_feeder_substations(feeder, role='train')

Return substation list for a specific feeder and role.

Parameters:

Name Type Description Default
feeder str

"feeder_A", "feeder_B", or "feeder_C".

required
role str

"train" or "zone_holdout".

'train'

Returns:

Type Description
List[str]

List of substation names for that feeder/role combination.

powerzoojax.data.ausgrid_utils.select_full_coverage_substations(df, start, end, *, max_gap_steps=2, substation_col='region', time_col='datetime', resolution='15min')

Return substations that have no consecutive time gap > max_gap_steps.

A gap is defined as a run of consecutive missing 15-min slots within [start, end]. Substations with any gap run longer than max_gap_steps are excluded.

Parameters:

Name Type Description Default
df DataFrame

DataFrame with at least time_col and substation_col columns. Multi-region format (one row per region per timestamp).

required
start str

ISO date string, inclusive window start.

required
end str

ISO date string, inclusive window end.

required
max_gap_steps int

Maximum tolerated consecutive missing steps.

2
substation_col str

Column name identifying the substation/region.

'region'
time_col str

Column name for timestamps.

'datetime'
resolution str

Expected time resolution for the full grid.

'15min'

Returns:

Type Description
List[str]

Sorted list of substation names with full (or near-full) coverage.

Non-stationary sampling

from powerzoojax.data import EpisodeConfig, NonstationarySampler, apply_drift

powerzoojax.data.nonstationary.EpisodeConfig

Bases: NamedTuple

Episode-level non-stationarity parameters.

powerzoojax.data.nonstationary.NonstationarySampler(total_steps, total_episodes=500, *, max_steps=48, drift_rate=0.0003, rolling_sigma_steps=720.0)

Generates non-stationary episode parameters for DSO task.

Usage::

sampler = NonstationarySampler(
    total_steps=10080,     # 7 months × 48 steps/day
    total_episodes=500,
)
cfg = sampler.sample(episode_idx=100, key=jax.random.PRNGKey(0))
# → EpisodeConfig(episode_start=2340, drift_factor=1.03)

The sampler supports four modes:

  • "train" — rolling center + drift (default)
  • "iid" — uniform random start, no drift (same window)
  • "drift_shock" — fixed high drift factor (OOD)
  • "fixed" — fixed start, no drift (for reproducible eval)

Parameters:

Name Type Description Default
total_steps int

Length of the full feeder shape array (in time steps). For Ausgrid training window (7 months @ 48 steps/day) ≈ 10,080.

required
total_episodes int

Expected total training episodes (for phase_k calc).

500
max_steps int

Episode length (default 48 = 24h @ 30min).

48
drift_rate float

Per-episode drift rate (default 0.0003 = +0.03%/episode).

0.0003
rolling_sigma_steps float

Gaussian σ for month-rolling start sampling, in time steps (default 720 = 15 days × 48 steps/day).

720.0

sample(episode_idx, key, *, mode='train')

Sample episode parameters.

Parameters:

Name Type Description Default
episode_idx int

Current episode index (0-based).

required
key Array

JAX PRNG key.

required
mode str

Sampling mode — "train", "iid", "drift_shock", or "fixed".

'train'

Returns:

Type Description
EpisodeConfig

EpisodeConfig(episode_start, drift_factor).

sample_batch(start_idx, batch_size, key, *, mode='train')

Sample a batch of episode configs.

Parameters:

Name Type Description Default
start_idx int

First episode index in the batch.

required
batch_size int

Number of episodes.

required
key Array

JAX PRNG key (split internally).

required
mode str

Sampling mode.

'train'

Returns:

Type Description
list

List of EpisodeConfig of length batch_size.

powerzoojax.data.nonstationary.apply_drift(load_profiles_p, load_profiles_q, drift_factor)

Scale load profiles by a drift factor.

Parameters:

Name Type Description Default
load_profiles_p ndarray

(T, n_bus) active load in p.u.

required
load_profiles_q ndarray

(T, n_bus) reactive load in p.u.

required
drift_factor float

Multiplicative scaling (e.g. 1.03 for +3% load growth).

required

Returns:

Type Description
Tuple[ndarray, ndarray]

Scaled (load_profiles_p, load_profiles_q).

DC microgrid profiles

from powerzoojax.data import (
    cycle_profile,
    make_synthetic_cpu_profile,
    make_synthetic_solar_profile,
    make_synthetic_outdoor_temp_profile,
    make_all_synthetic_profiles,
    load_workload_profiles,
    apply_ood_transform,
    VALID_SOURCES,
    VALID_OOD_SCENARIOS,
)

apply_ood_transform(params, scenario) produces the OOD splits used by the DC Microgrid benchmark. Valid scenarios: workload_swap, workload_shock, renewable_drought, cooling_stress, dg_derating, sla_tighten.

powerzoojax.data.dc_microgrid_profiles.cycle_profile(arr, episode_len, start_step=0)

Slice or cyclically tile arr to exactly episode_len steps.

Parameters:

Name Type Description Default
arr ndarray

Source array of shape (T,) (float32). If empty (T=0), returned as-is.

required
episode_len int

Desired output length.

required
start_step int

Starting offset into the cyclically-tiled sequence.

0

Returns:

Type Description
ndarray

Array of shape (episode_len,) with dtype preserved.

Rules
  • If T >= episode_len and start_step == 0: simple slice.
  • Otherwise: tile enough repetitions to cover start_step + episode_len, then slice.

powerzoojax.data.dc_microgrid_profiles.make_all_synthetic_profiles(episode_len=DC_EPISODE_LEN, steps_per_day=DC_STEPS_PER_DAY)

Return all three synthetic profiles as a dict.

powerzoojax.data.dc_microgrid_profiles.load_workload_profiles(source='google', episode_len=DC_EPISODE_LEN, start_step=0, data_dir=None, manifest_dir=None, strict=True, require_real_data=False)

Load DC workload profiles, with controlled fallback behaviour.

Parameters:

Name Type Description Default
source str

Data source key — "google", "azure", or "alibaba".

'google'
episode_len int

Number of steps in the output profiles (default 288).

DC_EPISODE_LEN
start_step int

Cyclic offset into the source profile (default 0).

0
data_dir Optional[str]

Override for the parquet directory (None → default).

None
manifest_dir Optional[str]

Override for the manifests directory (None → default).

None
strict bool

Schema / manifest error handling (default True).

  • strict=True (default): non-infra exceptions (schema mismatch, signal-resolution failure, …) are re-raised immediately.
  • strict=False: all exceptions warn + synthetic fallback.
True
require_real_data bool

Infrastructure error handling (default False).

  • False (default): FileNotFoundError and ImportError (parquet engine absent) → UserWarning + synthetic fallback. Suitable for generic env construction and development.
  • True: FileNotFoundError and ImportError are ALSO re-raised — the caller requires a real workload profile. Use for OOD scenarios (workload_swap, workload_shock) where using synthetic data would silently corrupt the experiment.
False

Returns:

Type Description
Dict[str, ndarray]

Dict with keys "cpu_profile", "solar_profile",

Dict[str, ndarray]

"outdoor_temp_profile" — all jnp.float32 arrays of shape

Dict[str, ndarray]

(episode_len,).

Notes

Real data (cpu_profile): loaded from the DC workload parquet file when available.

Real data (solar_profile): loaded from the GB generation-by-type solar trace when available.

Real data (price_profile): loaded from the GB MID market trace when available.

Deterministic adapter (outdoor_temp_profile): synthetic sine curve — no weather manifest exists yet.

powerzoojax.data.dc_microgrid_profiles.apply_ood_transform(params, scenario, *, drought_factor=0.2, temp_delta=5.0, dg_derating_factor=0.6, sla_slack=1.2, data_dir=None, manifest_dir=None)

Apply an OOD scenario transform, returning a modified DCMicrogridParams.

Parameters:

Name Type Description Default
params 'DCMicrogridParams'

Base params — should have non-None profiles for transforms that modify them. If profiles are None, a synthetic fallback is generated automatically.

required
scenario str

One of VALID_OOD_SCENARIOS.

required
drought_factor float

Solar scaling for renewable_drought (default 0.2).

0.2
temp_delta float

Temperature offset [°C] for cooling_stress (default 5.0).

5.0
dg_derating_factor float

DG capacity multiplier for dg_derating (0.6).

0.6
sla_slack float

New deadline slack for sla_tighten (default 1.2).

1.2
data_dir Optional[str]

Override data dir for workload_swap / workload_shock.

None
manifest_dir Optional[str]

Override manifest dir for workload_swap / workload_shock.

None

Returns:

Type Description
'DCMicrogridParams'

New DCMicrogridParams with the transform applied.

Raises:

Type Description
ValueError

Unknown scenario.