Dataset

Dataset: a named collection of Field s (+ optional asset collection).

A dataset can be built in code (fields passed directly) or loaded lazily from a Rebase-compatible Hugging Face repo via Dataset.from_manifest(), where the repo’s rebase.yaml declares each field’s parquet file and availability rule:

name: heftcom2024
description: Hybrid Energy Forecasting and Trading Competition 2024
fields:
  wind_power:
    path: data/wind_power.parquet
    kind: actual
    availability_lag: 1h
  dwd_nwp:
    path: data/dwd_nwp.parquet
    kind: forecast            # parquet already has (issue_time, valid_time) index

class emflow.data.dataset.Dataset(name: str, description: t.Optional[str] = None, collection: t.Optional['edm.Collection'] = None, fields: t.Dict[str, Field] = <factory>)

Bases: object

Named fields + optional energydatamodel asset collection.

fields maps name -> Field. The legacy data dict-of-frames view is kept as a convenience (dataset.data["wind_power"] returns the raw frame), but evaluation code must go through a DataFeed, never through data.

name: str

description: t.Optional[str] = None

collection: t.Optional['edm.Collection'] = None

fields: t.Dict[str, Field]

field(name: str) → Field

add(field: Field) → Dataset

property data: Dict[str, DataFrame]

Raw frames by field name (convenience view — not leak-safe).

property list_data

classmethod from_manifest(repo: str, token: str | None = None) → Dataset

Load a dataset from a Rebase-compatible repo’s rebase.yaml.

repo is an rb://dataset/<owner>/<name> URI (or bare <owner>/<name>, treated as a dataset repo). Field frames are read with pandas via fsspec, so rb:// caching and HF_TOKEN handling apply.