CLI
whest run
Run local evaluation for an estimator.
whest run
Run local evaluation for an estimator.
whest run [options]| Option | Default | Description |
|---|---|---|
--estimator | Path to estimator.py (see https://github.com/AIcrowd/whest-starterkit for starter files). | |
--class | Estimator class name to load from the estimator file (auto-detected if omitted). | |
--runner | 'local' | Execution backend: 'local'/'inprocess' run in-process; 'subprocess'/'server' run in an isolated subprocess (default: local). |
--n-mlps | Number of MLPs to evaluate. Default: 10 when --dataset is not provided; otherwise the full dataset size. Clamped to the dataset size when --dataset is set and --n-mlps exceeds it. | |
--detail | 'raw' | Report verbosity: 'raw' for a concise summary or 'full' for expanded per-MLP detail (default: raw). |
--profile | Collect and display per-MLP FLOP/budget profiling breakdowns in the report. | |
--show-diagnostic-plots | Include diagnostic plot panes in the rendered (non-JSON) report. | |
--format | Select output format: rich, plain, or json. | |
--json | Alias for --format json. | |
--dataset | Path to a baked dataset directory, or hf://owner/repo[@revision] for HF Hub. | |
--streaming | Stream the dataset from HF instead of downloading it. Iteration-only (no random access). Data is NOT cached — subsequent runs will re-fetch. Useful for small --n-mlps debugging runs. See docs/guides/datasets.md#streaming-mode. | |
--revision | HF Hub revision (tag or commit SHA) for --dataset. | |
--split | For multi-split datasets, the split to evaluate. Required when the dataset is multi-split; optional when single-split (defaults to the only split). | |
--flop-budget | Effective compute budget per MLP in FLOPs. Caps C_m = F_m + lambda*R_m, which under the default lambda of 0 is a pure FLOP cap. Always honored; any flop_budget stored in --dataset's metadata is ignored. Default: 2_199_023_255_552 (2**41). Earlier rounds: 272_000_000_000 (2.72e11) and 68_000_000_000 (6.8e10) for v1-warmup. | |
--lambda-flops-per-second | Price of one second of residual wall time, in FLOP-equivalents, for C_m = F_m + lambda*R_m. Default: 0 — residual time is not priced, it is gated by --residual-wall-time-limit, so C_m = F_m. To re-score a Phase 1 round pass 1e11 with --no-residual-wall-time-limit AND --wall-time-limit 60 (that round's cap). Must not be negative. | |
--n-samples | Ground-truth Monte-Carlo samples per MLP for --dataset-less runs (default: 200_000). Monte-Carlo error on a per-neuron mean falls as 1/N, so raise this for a quieter comparison and lower it for a faster loop. Ignored when --dataset supplies baked targets. | |
--debug | Show full Python tracebacks for errors instead of condensed messages. | |
--fail-fast | Stop on the first estimator error and let the raw Python traceback propagate (combine with --debug to show it). | |
--wall-time-limit | 120.0 | Wall-clock time limit per predict() call, in seconds (default: 120.0, the graded cap). Lower it to reproduce a tighter budget locally; exceeding it zeroes that MLP's predictions. |
--setup-timeout | 5.0 | Wall-clock time limit for the one-time setup() call, in seconds (default: 5.0, the graded cap). Exceeding it fails the whole submission rather than one MLP, so it is worth testing against. |
--residual-wall-time-limit | 0.4 | Wall-clock limit on non-flopscope time per predict() call, in seconds (default: 0.4, the graded cap). Residual time is plumbing — unpacking mlp, control flow around your fnp calls, assembling the result — not computation. Exceeding it zeroes that MLP's predictions. Raise it to debug under a profiler; lower it to leave headroom for a slower grader. |
--no-residual-wall-time-limit | Disable the residual wall-time gate entirely (no cap). Needed to re-score a Phase 1 round, which was priced with a lambda rate rather than gated; pair it with --lambda-flops-per-second 1e11 and --wall-time-limit 60. Overrides --residual-wall-time-limit. | |
--seed | Random seed for the run. Without --dataset, seeds both MLP generation and estimator setup. With --dataset, MLP seeds come from the dataset; this flag seeds estimator setup only. Default: omitted (ctx.seed defaults to 0; run_config.seed is null in the JSON output). | |
--max-threads | Limit BLAS to at most N CPU threads. |