Data¶
Setup-time facade for parquet-backed time series. For the architecture-level role, see Architecture → Data pipeline. Nothing in powerzoojax.data enters the compiled rollout.
Prefer load_jax_profiles(...) for env construction and load_signals(...) for inspection. load_data(...) covers raw-column layouts and direct table access; benchmark code standardizes on load_jax_profiles(...) and the parquet facade documented in Architecture → Data pipeline.
Semantic loading¶
from powerzoojax.data import DataLoader, signals as S
loader = DataLoader()
profiles = loader.load_jax_profiles(
[S.LOAD_ACTUAL_MW, S.SOLAR_AVAILABLE_MW],
source="gb",
start_date="2025-04-01",
end_date="2025-12-31",
resample="30min",
)
Stable signal names¶
Power-system signals:
LOAD_ACTUAL_MWLOAD_FORECAST_DA_MWLOAD_FORECAST_P10_MWLOAD_FORECAST_P50_MWLOAD_FORECAST_P90_MWSOLAR_AVAILABLE_MWWIND_AVAILABLE_MW
Market index signals:
MARKET_MID_PRICE_APXMARKET_MID_PRICE_N2EXMARKET_MID_VOLUME_APXMARKET_MID_VOLUME_N2EX
Data-center signals:
DC_CPU_UTILDC_MEM_UTILDC_NET_INDC_NET_OUTDC_DISK_IODC_POWER_MWDC_GPU_UTILDC_GPU_MEM_UTIL
Loader, manifest, registry¶
powerzoojax.data.data_loader.DataLoader(data_dir=None, manifest_dir=None)
¶
Benchmark data facade backed by parquet files and manifest metadata.
Features:
- JAX array output via :meth:load_jax_profiles (float32, GPU-ready)
- Semantic signal loading via :meth:load_signals
- Calendar / profile time-alignment across heterogeneous sources
- Forecast-panel loading with issue_time / target_time
- Legacy column-based loading via :meth:load_data
- Resampling, date filtering, multi-dataset merge
Initialize DataLoader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_dir
|
Optional[Union[str, Path]]
|
Directory containing parquet files.
If None, defaults to |
None
|
manifest_dir
|
Optional[Union[str, Path]]
|
Directory containing manifest JSON files.
If None, defaults to |
None
|
registry
property
¶
Expose the dataset registry for advanced querying.
load_jax_profiles(signals, *, source=None, region=None, start_date=None, end_date=None, resample=None, time_alignment=None, interpolation='linear', dtype=jnp.float32)
¶
Load semantic signals and return as a JAX array.
This is the primary entry-point for env construction. It calls
:meth:load_signals internally, then converts numeric columns to
a jnp.ndarray of shape (T, n_signals).
The result is a JAX array on the default device (GPU when available),
ready to be passed to make_trans_params(case, load_profiles=...).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
signals
|
List[str]
|
Semantic signal names (e.g.
|
required |
source
|
Optional[str]
|
Restrict lookup to a specific data source. |
None
|
region
|
Optional[str]
|
Filter by region (e.g. |
None
|
start_date
|
Optional[Union[str, datetime, date]]
|
Simulation time window start (inclusive). |
None
|
end_date
|
Optional[Union[str, datetime, date]]
|
Simulation time window end (inclusive). |
None
|
resample
|
Optional[str]
|
Target frequency ( |
None
|
time_alignment
|
Optional[Dict[str, str]]
|
Per-signal calendar-shift overrides. |
None
|
interpolation
|
str
|
Interpolation method when resampling. |
'linear'
|
dtype
|
dtype
|
JAX dtype for the output array. Default |
float32
|
Returns:
| Type | Description |
|---|---|
ndarray
|
|
ndarray
|
ordered as signals. |
load_jax_array(df, columns=None, dtype=jnp.float32)
¶
Convert a DataFrame (e.g. from :meth:load_signals) to JAX.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Source DataFrame. |
required |
columns
|
Optional[List[str]]
|
Columns to extract (order preserved). If None, all numeric columns are used. |
None
|
dtype
|
dtype
|
Target JAX dtype. |
float32
|
Returns:
| Type | Description |
|---|---|
ndarray
|
|
load_signals(signals, *, source=None, region=None, start_date=None, end_date=None, resample=None, time_alignment=None, interpolation='linear')
¶
Load one or more semantic signals into a single DataFrame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
signals
|
List[str]
|
Semantic signal names (e.g.
|
required |
source
|
Optional[str]
|
Restrict lookup to a specific data source
( |
None
|
region
|
Optional[str]
|
Filter by region (e.g. |
None
|
start_date
|
Optional[Union[str, datetime, date]]
|
Simulation time window start (inclusive). |
None
|
end_date
|
Optional[Union[str, datetime, date]]
|
Simulation time window end (inclusive). |
None
|
resample
|
Optional[str]
|
Target frequency ( |
None
|
time_alignment
|
Optional[Dict[str, str]]
|
Per-signal calendar-shift overrides. |
None
|
interpolation
|
str
|
Interpolation method when resampling. |
'linear'
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame indexed by |
load_actual_series(signals, **kwargs)
¶
Convenience wrapper: load only actual_series data.
load_forecast_panel(signals, *, source=None, region=None, start_date=None, end_date=None)
¶
Load forecast-panel data with issue_time + target_time.
list_available_datasets()
¶
List all available datasets (parquet files).
get_metadata(dataset_name)
¶
Get metadata for a dataset.
get_available_columns(dataset_name)
¶
Get available columns for a dataset.
get_date_range(dataset_name)
¶
Get date range for a dataset.
find_datasets_for_columns(columns)
¶
Find which datasets contain the requested columns.
load_data(dataset_name=None, columns=None, start_date=None, end_date=None, validate_dates=True, merge_how='inner', resample=None, interpolation='linear')
¶
Load data from parquet file(s) with optional resampling.
If dataset_name is not provided, automatically finds datasets based on columns.
print_dataset_info(dataset_name)
¶
Print information about a dataset.
get_summary()
¶
Get summary of currently loaded data.
powerzoojax.data.manifest.DatasetManifest(name, source, data_type, time_mode, resolution, parquet_file, column_map=dict(), index_map=dict(), derived=dict(), normalize=dict(), data_epoch=None, cyclical=False, region_values=list(), date_range=None, metadata_json=None, source_url=None, source_urls=None, source_organization=None)
dataclass
¶
Declarative description of one dataset.
A manifest is the single source of truth that tells the data layer: * which parquet file to read, * how to map raw columns to semantic signals, * how to handle time alignment (calendar vs. profile), and * whether data can be tiled cyclically.
powerzoojax.data.registry.DatasetRegistry(manifest_dir=None)
¶
Load all manifest JSONs and build a reverse index from signal -> datasets.
powerzoojax.data.alignment.TimeAligner
¶
Map raw-data timestamps onto a unified simulation timeline.
align_calendar(df, sim_start, sim_end, align_from=None, time_col='datetime')
staticmethod
¶
Shift a calendar-mode dataset onto [sim_start, sim_end].
align_profile(df, sim_start, sim_end, resolution, time_col='datetime')
staticmethod
¶
Tile a profile cyclically to cover [sim_start, sim_end].
Uses jnp.tile so the tiled numeric data stays on the JAX
default device (GPU when available).
tile_profile_jax(values, n_needed)
staticmethod
¶
align(df, *, time_mode, sim_start, sim_end, align_from=None, resolution='30min', time_col='datetime')
classmethod
¶
Dispatch to the correct alignment strategy.
Frozen splits¶
from powerzoojax.data import (
GB_TRAIN_START, GB_TRAIN_END,
GB_IID_START, GB_IID_END,
AUSGRID_TRAIN_START, AUSGRID_TRAIN_END,
AUSGRID_IID_START, AUSGRID_IID_END,
AUSGRID_SUMMER_START, AUSGRID_SUMMER_END,
gb_windows,
ausgrid_windows,
)
powerzoojax.data.splits.gb_windows()
¶
Return ((train_start, train_end), (iid_start, iid_end)).
powerzoojax.data.splits.ausgrid_windows()
¶
Return (train_window, iid_window, summer_ood_window).
Ausgrid utilities¶
from powerzoojax.data import (
AUSGRID_FEEDER_POOLS,
AUSGRID_TRAIN_POOL,
AUSGRID_ZONE_HOLDOUT_POOL,
get_ausgrid_split,
get_feeder_substations,
select_full_coverage_substations,
)
powerzoojax.data.ausgrid_utils.get_ausgrid_split(role)
¶
Return (start, end, substation_list) for a named split role.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
role
|
str
|
One of |
required |
Returns:
| Type | Description |
|---|---|
Tuple[str, str, List[str]]
|
Tuple of (ISO-date start, ISO-date end, list of substation names). |
Raises:
| Type | Description |
|---|---|
KeyError
|
if role is not recognised. |
powerzoojax.data.ausgrid_utils.get_feeder_substations(feeder, role='train')
¶
Return substation list for a specific feeder and role.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feeder
|
str
|
|
required |
role
|
str
|
|
'train'
|
Returns:
| Type | Description |
|---|---|
List[str]
|
List of substation names for that feeder/role combination. |
powerzoojax.data.ausgrid_utils.select_full_coverage_substations(df, start, end, *, max_gap_steps=2, substation_col='region', time_col='datetime', resolution='15min')
¶
Return substations that have no consecutive time gap > max_gap_steps.
A gap is defined as a run of consecutive missing 15-min slots within
[start, end]. Substations with any gap run longer than
max_gap_steps are excluded.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
DataFrame with at least |
required |
start
|
str
|
ISO date string, inclusive window start. |
required |
end
|
str
|
ISO date string, inclusive window end. |
required |
max_gap_steps
|
int
|
Maximum tolerated consecutive missing steps. |
2
|
substation_col
|
str
|
Column name identifying the substation/region. |
'region'
|
time_col
|
str
|
Column name for timestamps. |
'datetime'
|
resolution
|
str
|
Expected time resolution for the full grid. |
'15min'
|
Returns:
| Type | Description |
|---|---|
List[str]
|
Sorted list of substation names with full (or near-full) coverage. |
Non-stationary sampling¶
powerzoojax.data.nonstationary.EpisodeConfig
¶
Bases: NamedTuple
Episode-level non-stationarity parameters.
powerzoojax.data.nonstationary.NonstationarySampler(total_steps, total_episodes=500, *, max_steps=48, drift_rate=0.0003, rolling_sigma_steps=720.0)
¶
Generates non-stationary episode parameters for DSO task.
Usage::
sampler = NonstationarySampler(
total_steps=10080, # 7 months × 48 steps/day
total_episodes=500,
)
cfg = sampler.sample(episode_idx=100, key=jax.random.PRNGKey(0))
# → EpisodeConfig(episode_start=2340, drift_factor=1.03)
The sampler supports four modes:
"train"— rolling center + drift (default)"iid"— uniform random start, no drift (same window)"drift_shock"— fixed high drift factor (OOD)"fixed"— fixed start, no drift (for reproducible eval)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
total_steps
|
int
|
Length of the full feeder shape array (in time steps). For Ausgrid training window (7 months @ 48 steps/day) ≈ 10,080. |
required |
total_episodes
|
int
|
Expected total training episodes (for phase_k calc). |
500
|
max_steps
|
int
|
Episode length (default 48 = 24h @ 30min). |
48
|
drift_rate
|
float
|
Per-episode drift rate (default 0.0003 = +0.03%/episode). |
0.0003
|
rolling_sigma_steps
|
float
|
Gaussian σ for month-rolling start sampling, in time steps (default 720 = 15 days × 48 steps/day). |
720.0
|
sample(episode_idx, key, *, mode='train')
¶
Sample episode parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
episode_idx
|
int
|
Current episode index (0-based). |
required |
key
|
Array
|
JAX PRNG key. |
required |
mode
|
str
|
Sampling mode — |
'train'
|
Returns:
| Type | Description |
|---|---|
EpisodeConfig
|
|
sample_batch(start_idx, batch_size, key, *, mode='train')
¶
Sample a batch of episode configs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
start_idx
|
int
|
First episode index in the batch. |
required |
batch_size
|
int
|
Number of episodes. |
required |
key
|
Array
|
JAX PRNG key (split internally). |
required |
mode
|
str
|
Sampling mode. |
'train'
|
Returns:
| Type | Description |
|---|---|
list
|
List of |
powerzoojax.data.nonstationary.apply_drift(load_profiles_p, load_profiles_q, drift_factor)
¶
Scale load profiles by a drift factor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
load_profiles_p
|
ndarray
|
(T, n_bus) active load in p.u. |
required |
load_profiles_q
|
ndarray
|
(T, n_bus) reactive load in p.u. |
required |
drift_factor
|
float
|
Multiplicative scaling (e.g. 1.03 for +3% load growth). |
required |
Returns:
| Type | Description |
|---|---|
Tuple[ndarray, ndarray]
|
Scaled |
DC microgrid profiles¶
from powerzoojax.data import (
cycle_profile,
make_synthetic_cpu_profile,
make_synthetic_solar_profile,
make_synthetic_outdoor_temp_profile,
make_all_synthetic_profiles,
load_workload_profiles,
apply_ood_transform,
VALID_SOURCES,
VALID_OOD_SCENARIOS,
)
apply_ood_transform(params, scenario) produces the OOD splits used by the DC Microgrid benchmark. Valid scenarios: workload_swap, workload_shock, renewable_drought, cooling_stress, dg_derating, sla_tighten.
powerzoojax.data.dc_microgrid_profiles.cycle_profile(arr, episode_len, start_step=0)
¶
Slice or cyclically tile arr to exactly episode_len steps.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
arr
|
ndarray
|
Source array of shape |
required |
episode_len
|
int
|
Desired output length. |
required |
start_step
|
int
|
Starting offset into the cyclically-tiled sequence. |
0
|
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape |
Rules
- If
T >= episode_lenandstart_step == 0: simple slice. - Otherwise: tile enough repetitions to cover
start_step + episode_len, then slice.
powerzoojax.data.dc_microgrid_profiles.make_all_synthetic_profiles(episode_len=DC_EPISODE_LEN, steps_per_day=DC_STEPS_PER_DAY)
¶
Return all three synthetic profiles as a dict.
powerzoojax.data.dc_microgrid_profiles.load_workload_profiles(source='google', episode_len=DC_EPISODE_LEN, start_step=0, data_dir=None, manifest_dir=None, strict=True, require_real_data=False)
¶
Load DC workload profiles, with controlled fallback behaviour.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
Data source key — |
'google'
|
episode_len
|
int
|
Number of steps in the output profiles (default 288). |
DC_EPISODE_LEN
|
start_step
|
int
|
Cyclic offset into the source profile (default 0). |
0
|
data_dir
|
Optional[str]
|
Override for the parquet directory ( |
None
|
manifest_dir
|
Optional[str]
|
Override for the manifests directory ( |
None
|
strict
|
bool
|
Schema / manifest error handling (default
|
True
|
require_real_data
|
bool
|
Infrastructure error handling (default
|
False
|
Returns:
| Type | Description |
|---|---|
Dict[str, ndarray]
|
Dict with keys |
Dict[str, ndarray]
|
|
Dict[str, ndarray]
|
|
Notes
Real data (cpu_profile): loaded from the DC workload parquet file when available.
Real data (solar_profile): loaded from the GB generation-by-type solar trace when available.
Real data (price_profile): loaded from the GB MID market trace when available.
Deterministic adapter (outdoor_temp_profile): synthetic sine curve — no weather manifest exists yet.
powerzoojax.data.dc_microgrid_profiles.apply_ood_transform(params, scenario, *, drought_factor=0.2, temp_delta=5.0, dg_derating_factor=0.6, sla_slack=1.2, data_dir=None, manifest_dir=None)
¶
Apply an OOD scenario transform, returning a modified DCMicrogridParams.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
params
|
'DCMicrogridParams'
|
Base params — should have non-None profiles for transforms that modify them. If profiles are None, a synthetic fallback is generated automatically. |
required |
scenario
|
str
|
One of |
required |
drought_factor
|
float
|
Solar scaling for |
0.2
|
temp_delta
|
float
|
Temperature offset [°C] for |
5.0
|
dg_derating_factor
|
float
|
DG capacity multiplier for |
0.6
|
sla_slack
|
float
|
New deadline slack for |
1.2
|
data_dir
|
Optional[str]
|
Override data dir for workload_swap / workload_shock. |
None
|
manifest_dir
|
Optional[str]
|
Override manifest dir for workload_swap / workload_shock. |
None
|
Returns:
| Type | Description |
|---|---|
'DCMicrogridParams'
|
New |
Raises:
| Type | Description |
|---|---|
ValueError
|
Unknown scenario. |