Verifier

Verifier: Experiment + untrusted-submission policy.

The evaluation loop itself is the ordinary Experiment — leakage protection lives in the data layer (every observation is a feed-served view), not in a special runner. What the Verifier adds is submission hygiene for untrusted models:

• the submitter provides a fresh, untrained Predictor (directly or via get_model()); the verifier fits it itself on the official training view, so it can never have been fit on test targets;

• scoring runs on the holdout split, which agents must never iterate on (they hillclimb on validation);

• the score is computed by the verifier from the returned predictions — nothing the submission prints is trusted;

• results append to a leaderboard CSV, and when the problem carries published reference scores the scorecard reports where the submission would have placed in the original competition.

Honest scope note: for problems whose full dataset ships locally (the tutorial problems), a determined adversary can read the raw files outside the feed — local verification protects against accidental leakage. The trustworthy setup for agent benchmarking keeps holdout labels in a private rb:// repo that only the verifier’s token can read, and runs submissions with no network.

class emflow.run.verifier.Verifier(problem: str | Problem = 'swedish-temperatures:ar', leaderboard_path=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/emflow/checkouts/latest/submissions/leaderboard.csv'), split: str = 'holdout')

Bases: object

Train and strictly evaluate a submitted predictor on a problem’s holdout.

verify(submission, name=None, record=True, verbose=True, mode: str = 'auto', metadata: dict | None = None) → Result

Evaluate a submission and return its Result.

submission is a fresh Predictor or a zero-arg factory returning one (the get_model() convention).

metadata (optional, str/number values) is recorded verbatim on the leaderboard row and echoed on the scorecard. This is the seam for search harnesses to make performance claims honest about selection: rebase-hillclimb passes e.g. {"n_trials": N} from its run journal, and any deflation/PBO math lives there — emflow only records it.