Problem

Problem: the complete, self-describing benchmark bundle.

load_problem("heftcom2024:forecasting") returns one of these. It carries everything an Experiment needs — dataset, environment factory, objective, schedule, splits — plus the published reference scores that let a result be ranked against the historical field.

class emflow.problems.problem.Splits(train_end: Timestamp, validation: Tuple[Timestamp, Timestamp], holdout: Tuple[Timestamp, Timestamp])

Bases: object

Temporal splits, in target time.

train_end

Training data is everything strictly before this timestamp.

validation

Half-open [start, end) period whose origins are for model iteration — what agents hillclimb on.

holdout

Half-open [start, end) period scored once by the Verifier. Never iterate against it.

train_end: Timestamp

validation: Tuple[Timestamp, Timestamp]

holdout: Tuple[Timestamp, Timestamp]

period(split: str) → Tuple[Timestamp, Timestamp]

class emflow.problems.problem.RefScore(rank: int, team: str, score: float)

Bases: object

One row of a competition’s published final leaderboard.

rank: int

team: str

score: float

class emflow.problems.problem.Problem(name: str, dataset: t.Union[Dataset, str, t.Callable[[], Dataset]], make_env: t.Callable[['Problem', str], 'gym.Env'], objective: Objective, schedule: IssueSchedule, splits: Splits, description: t.Optional[str] = None, reference_scores: t.List[RefScore] = <factory>, _dataset_cache: t.Optional[Dataset] = None)

Bases: object

A benchmark problem. Environments are created fresh per run via env() — never share a live env between runs.

name: str

dataset: t.Union[Dataset, str, t.Callable[[], Dataset]]

make_env: t.Callable[['Problem', str], 'gym.Env']

objective: Objective

schedule: IssueSchedule

splits: Splits

description: t.Optional[str] = None

reference_scores: t.List[RefScore]

load_dataset() → Dataset

Resolve the dataset (lazily, cached): a Dataset instance, an rb:// manifest reference, or a zero-arg loader callable.

env(split: str = 'validation') → gym.Env

Build a fresh environment over the given split’s origins.

origins(split: str = 'validation')

rank_of(score: float) → int | None

1-based position score would have taken on the published leaderboard (None if no reference scores).