whestbench.
Participant GuideHow to

Validate, Run, and Package

Sourced from whest-starterkit @ 9bfb6df.

Validate, Run, and Package

← Documentation

🎯 When to use this page

Use this page for the standard local participant loop.

πŸš€ Do this now

Validate estimator loading and output contract:

whest validate is a fast sanity check using a small fixed MLP (width=4, depth=2). It verifies loading, output shape, and value finiteness β€” not full behavioral or performance correctness. Always follow with whest run for realistic tests.

whest validate --estimator estimator.py

Run local scoring (recommended default runner):

whest run --estimator estimator.py

whest run defaults to --runner local for fast iteration.

Run against the published evaluation dataset on HuggingFace (skips sampling β€” much faster for repeated runs, no local bake needed):

whest run \
    --estimator estimator.py \
    --dataset hf://aicrowd/arc-whestbench-public-2026@v1-phase1
# auto-resolves to the `mini` split (100 MLPs, ~850 MB cached after first call)

Or bake a custom local dataset once and reuse it:

whest dataset bake --output ./my-eval --n-mlps 10 --n-samples 10000
whest run --estimator estimator.py --dataset ./my-eval

See Use Evaluation Datasets for details.

Run faster local debug path:

whest run --estimator estimator.py --runner local

Run with machine-readable output:

whest run --estimator estimator.py --runner local --format json

--json still works as an alias, but --format rich|plain|json is the canonical output selector across the CLI.

Package submission artifact (single file β€” the common case):

whest package --estimator estimator.py --output ./submission.tar.gz

A file argument ships only that file; a folder argument (--estimator .) ships every file in that folder. Either way whest package previews exactly what will be submitted and (in folder mode) asks for confirmation before writing β€” pass --yes / -y to skip the prompt in CI or scripts. Credential files (.env, *.pem, keys, …) are never included. The 50 MiB / 50-file caps apply; use .whestignore to exclude scratch or large artefacts.

Shipping helper modules or precomputed weights? Keep them in the folder and package the folder β€” they ship by being present, no extra flags. (Third-party PyPI packages can't be shipped β€” the grader installs nothing beyond flopscope and the whestbench API, so do that work offline and ship the result as data.) See Ship Weights and Multi-File Submissions.

Useful whest run flags

These all show up in whest run --help but get lost there. Reach for them when:

FlagReach for it when…
--seed NDeterministic comparison between two estimator versions. Pin the seed and the same MLPs, the same per-MLP mlp.seed values, and the same SetupContext.seed are used across runs. Also accepted by whest validate (seeds the validation setup(ctx) call). With --dataset, the dataset supplies the per-MLP seeds and --seed controls ctx.seed only.
--n-samples NGround-truth sampling samples per MLP. The contest default (in whest run without an explicit override) is 100 * 100 * 256 = 2,560,000; whest dataset bake --n-samples defaults to 10000. Drop to --n-samples 5000 for a ~10x faster local sanity check; raise back up before drawing real conclusions.
--n-mlps NDefault 10. Drop to 3 while iterating to halve runtime; raise to 20+ when you're trying to reduce noise on a close score.
--flop-budget NDefault 2.72e11 (the phase-1 grader effective-compute budget β€” caps C_m = F_m + λ·R_m, not just analytical FLOPs; the v1-warmup round used 6.8e10). Bump to 1e12 to confirm an algorithm idea isn't budget-starved before optimizing for budget.
--profileEmits a per-namespace FLOP/time breakdown so you can see where your estimator burns the budget.
--show-diagnostic-plotsRenders convergence and per-layer error plots inline (terminal-friendly). Pairs well with --profile.
--max-threads NPin the BLAS thread pool size so wall_time_s is comparable across machines. Useful when triaging a "fast on my laptop, slow in CI" report.
--detail {raw,full}raw strips Rich formatting (handy for tee-ing logs); full adds the per-MLP raw arrays.
--wall-time-limit SCap each predict() call's wall time. Useful when local debugging hangs on a numerical edge case.
--residual-wall-time-limit SCap time spent outside flopscope ops (Python plumbing, loops, control flow). Surfaces "looks fast in FLOPs but Python is the bottleneck" issues.
--debug + --fail-fastFirst exception β†’ halt + raw traceback. Combine for the fastest "what broke?" loop.

βœ… Expected outcome

  • validate passes,
  • run produces a score report,
  • package creates a .tar.gz artifact.

⚠️ Common first failure

Symptom: run fails after validate passed.

Use this escalation flow:

  1. Retry with debug info:
whest run --estimator estimator.py --debug
  1. If traceback still feels opaque, rerun in-process:
whest run --estimator estimator.py --runner local --debug

For runner modes, see Stage 3: Run Locally, Stage 4: Subprocess Runner, and the Debugging Checklist.

Concrete example:

Error [predict:PREDICT_ERROR]: Estimator predict failed.
Use --debug to include a traceback.
Tip: For estimator-level tracebacks, rerun with --runner local --debug.

Next command to run:

whest run --estimator estimator.py --runner local --debug

➑️ Next step

Use Evaluation Datasets

Previous Page

Write an Estimator

Next Page

On this page

Validate, Run, and Package🎯 When to use this pageπŸš€ Do this nowUseful whest run flagsβœ… Expected outcome⚠️ Common first failure➑️ Next step