Participant GuideGetting started
Tutorial — The 5-stage ladder
Sourced from whest-starterkit @
aaa3882.
Tutorial — The 5-stage ladder
The tutorial trail. Each stage is a single command on the same estimator.py, with the harness adding one more level of formality at each step. Read top-to-bottom: Stage 4's subprocess isolation catches bugs that Stage 3 hides, so skipping ahead is rarely worth it.
| Stage | Command | What it adds | Doc |
|---|---|---|---|
| 1 | uv run python estimator.py | The math. Iterate locally with flopscope and local_engine.py; no whest CLI required. | stage-1-standalone.md |
| 2 | uv run whest validate --estimator estimator.py | Contract correctness — class resolved, optional setup() runs, shape, finite values. | stage-2-validate.md |
| 3 | uv run whest run --estimator estimator.py --dataset hf://aicrowd/arc-whestbench-public-2026 --split mini --runner local | Real scoring against the public Mini split (100 MLPs), in-process (so pdb works). | stage-3-run-local.md |
| 4 | uv run whest run --estimator estimator.py --dataset hf://aicrowd/arc-whestbench-public-2026 --split mini --runner subprocess | Subprocess isolation — catches state-bleed between MLPs, dirty imports, RNG re-use. | stage-4-run-subprocess.md |
| 5 | uv run whest package --estimator estimator.py --output submission.tar.gz | Package the submission tarball for AIcrowd. | stage-5-package.md |
Each stage doc carries an "Expected outcome" callout so you know what success looks like before climbing — and a "Ladder" strip at the top so you always know where you are.
➡️ Where to look next
- Ready to ship? → Stage 5 → Submit to AIcrowd (
whest loginthenwhest submit). - Got a working estimator and want a better score? → How-to: algorithm ideas, Reference: code patterns.
- Score regressed after a change? → How-to: debugging checklist, Troubleshooting.
- Need the exact contract? → Reference: estimator contract.
- Sanity-check before clicking "submit"? → How-to: pre-submission checklist.