Sourced from whest-starterkit @ 9bfb6df.

Common Participant Errors

← Documentation

Use this page when validate or run fails.

Understand runner modes first

whest run --estimator ... uses --runner local by default.

• local (default): in-process execution with best traceback fidelity while debugging.
• subprocess: isolated process execution for stricter reproduction; server remains a legacy alias.

Fast debug ladder:

whest run --estimator estimator.py
whest run --estimator estimator.py --debug
whest run --estimator estimator.py --runner local --debug

Sample server-style failure:

Error [setup:SETUP_ERROR]: Estimator setup failed.
Use --debug to include a traceback.
Tip: For estimator-level tracebacks, rerun with --runner local --debug.

Exact follow-up:

whest run --estimator estimator.py --runner local --debug

Estimator returned wrong shape

Symptom: error mentions expected shape (depth, width).

Why it happens: returned wrong dimensions or a 1D array.

Fix now: ensure predict returns a flopscope array with shape (mlp.depth, mlp.width). Use fnp.zeros((mlp.depth, mlp.width)) as a starting point.

Verify:

whest validate --estimator estimator.py

Non-finite values (nan or inf)

Symptom: error mentions finite values.

Why it happens: unstable numeric operations.

Fix now: add guards/clipping/checks in your prediction logic.

Verify:

whest validate --estimator estimator.py

FLOP budget exceeded

Symptom: unexpectedly poor adjusted_final_layer_score despite reasonable prediction logic, with one or more MLPs showing budget_exhausted: true or combined_budget_exhausted: true.

Why it happens: your estimator's effective compute C_m = F_m + λ·R_m exceeded flop_budget. The affected MLP's predictions are replaced with zeros and the per-MLP multiplier is forced to 1.0 (no compute discount), so adjusted_final_layer_score_m = MSE(0, Y_m) × 1.0 — strictly worse than a trivial-zero submission that succeeds (which gets the 0.1 multiplier floor).

budget_exhausted fires when flopscope itself trips (your analytical FLOPs exceed the cap). combined_budget_exhausted fires on the post-hoc check C_m > B — flopscope didn't trip, but the residual-wall-time penalty (λ · residual_wall_time_s, λ default 1e11 FLOPs/sec) pushed effective compute past the cap.

Fix now:

• check flops_used, effective_compute, residual_wall_time_s, budget_exhausted, and combined_budget_exhausted in the per-MLP report,
• reduce expensive operations (matmul dominates FLOP cost),
• reduce Python-side overhead — tight loops over neurons add to residual_wall_time_s and thus to effective_compute,
• consider diagonal approximations instead of full covariance,
• see Manage Your FLOP Budget for optimization guidance.

Verify:

whest run --estimator estimator.py --json

Class not found

Symptom: "No estimator class found" or ImportError.

Why it happens: your class must be named Estimator (or specify --class).

Fix now: rename your class to Estimator.

Verify:

whest validate --estimator estimator.py

Import error in estimator

Symptom: ModuleNotFoundError when loading your file.

Why it happens: your estimator imports something the grader sandbox doesn't provide. At grading time only flopscope (incl. flopscope.numpy as fnp), the whestbench API (BaseEstimator, MLP, SetupContext), and the Python standard library are importable — there is no requirements.txt install step, so third-party packages (numpy, scipy, torch, …) are not available. (A missing helper module is different: it ships if you package the folder — --estimator . — instead of the single file.)

Fix now: for all array math use import flopscope as flops and import flopscope.numpy as fnp (not import numpy). Ship multi-file estimators as a folder. For work that genuinely needs a third-party library (a PyTorch-trained model, a scipy routine), compute it offline before packaging and ship the result as a pickle-free .npz, loaded in setup() — see Ship Weights. Note: whest validate runs in your local venv (which has numpy/scipy/torch), so it will not reproduce a grader-only missing-package error — the only fix is to not import those packages.

Verify:

whest validate --estimator estimator.py

Signature mismatch

Symptom: TypeError: predict() missing 1 required positional argument.

Why it happens: your predict method has the wrong signature.

Fix now: ensure signature is def predict(self, mlp: MLP, budget: int) -> fnp.ndarray:.

Verify:

whest validate --estimator estimator.py

Predict raised an unexpected exception

Symptom: whest run exits with status 1 and prints an "Estimator Errors" panel listing one or more MLPs with a PREDICT_ERROR code. A stderr line reads e.g. 2 of 10 MLP(s) raised during predict; rerun with --debug for tracebacks....

Why it happens: your predict() raised an exception that is neither BudgetExhaustedError nor TimeExhaustedError. WhestBench routes the failure through the zero-prediction path — the affected MLP scores final_layer_mse_m × 1.0 (no compute discount) and the suite mean stays finite. The non-zero exit code signals that the submission is not yet passing.

Fix now:

# Show full tracebacks in the "Estimator Errors" panel:
whest run --estimator estimator.py --debug

# Stop at the first failure and propagate the raw Python traceback:
whest run --estimator estimator.py --debug --fail-fast

The traceback in the panel (or the raw stack from --fail-fast) points directly at the line in your estimator that raised.

Setup timeout

Symptom: SETUP_TIMEOUT error.

Why it happens: setup() exceeded the time limit (typically 5 seconds).

Fix now: move expensive computation from setup() to predict(), or reduce setup work.

Verify:

whest run --estimator estimator.py --runner local --debug

Predict timeout

Symptom: PREDICT_TIMEOUT error.

Why it happens: predict() exceeded the wall-clock safety limit.

Fix now: check for infinite loops or extremely expensive operations. This is a safety guardrail, not the FLOP budget.

Verify:

whest run --estimator estimator.py --runner local --debug

Budget exhausted mid-operation

Symptom: BudgetExhaustedError raised during a specific operation.

Why it happens: a single flopscope operation would exceed your remaining FLOP budget.

Fix now: use flops.budget_summary() to find the expensive operation. Consider diagonal approximations or fewer iterations.

Verify: check flops_used in the score report.

Numerical instability in deep networks

Symptom: predictions become nan or inf after many layers.

Why it happens: values grow or shrink exponentially through deep networks without safeguards.

Fix now: add overflow guards — rescale covariance when diagonal values exceed a threshold (see covariance_propagation.py example). Use float64 for intermediate calculations.

Verify:

whest validate --estimator estimator.py

Dtype mismatch

Symptom: output is float64 but evaluator expects float32, or similar type issues.

Why it happens: flopscope operations may produce different dtypes than expected.

Fix now: cast your output: return fnp.asarray(result, dtype=fnp.float32).

Verify:

whest validate --estimator estimator.py

Empty predictions

Symptom: returned shape (0, width) or similar zero-length array.

Why it happens: your layer loop did not iterate (empty mlp.weights).

Fix now: check that you iterate over mlp.weights and append results per layer.

Verify:

whest validate --estimator estimator.py

Using numpy instead of flopscope

Symptom: operations work but FLOP budget is not consumed (shows 0 flops_used).

Why it happens: you are using import numpy as np instead of import flopscope.numpy as fnp. Numpy operations are not FLOP-tracked. (This is the local symptom — your venv has numpy, so it runs silently untracked. On the grader import numpy fails outright, since the sandbox has no numpy — see Import error in estimator. Either way, np.* is wrong; use fnp.*.)

Fix now: replace all np.* calls with fnp.* equivalents. See Code Patterns.

Verify: check flops_used > 0 in score report.

Every MLP failed (n_failed_mlps == n_mlps)

Symptom: the suite-level failure_breakdown shows every MLP carrying at least one failure flag (n_failed_mlps == n_mlps), and the adjusted_final_layer_score is dominated by MSE(0, Y_m) × 1.0 across the board (typically lands near 0.91, the raw final_layer_mse of zero predictions at the default activation scale).

Note: this is the post-merge replacement for the older "score is inf" symptom. Since whestbench PR #39 (May 2026) failures produce finite scores at the zero-prediction × 1.0 multiplier — there is no longer an inf path.

Why it happens: every MLP either raised during predict() or exhausted the FLOP / wall-time / residual-wall-time / combined budget.

Tell them apart from failure_breakdown and the exit code:

• Exit 1 + non-zero failure_breakdown.error + "Estimator Errors" panel — predict() raised exceptions on at least one MLP. See Predict raised an unexpected exception.
• Exit 0 + every per_mlp[i].budget_exhausted: true — you ran out of analytical FLOPs.
• Exit 0 + every per_mlp[i].combined_budget_exhausted: true — your effective_compute = F_m + λ·R_m exceeded the cap (residual wall time pushed you over, even though flopscope didn't trip).
• Exit 0 + every per_mlp[i].time_exhausted: true — you ran out of wall-clock time.

Fix now: run with --debug to see tracebacks in the "Estimator Errors" panel (works with any runner), or --fail-fast to halt at the first failing MLP with the raw Python stack:

whest run --estimator estimator.py --debug
whest run --estimator estimator.py --debug --fail-fast

Setup runs expensive operations

Symptom: unexpected FLOP usage or budget consumption before predict().

Why it happens: setup() runs outside any BudgetContext, so flopscope operations there use the default (very large) budget. This is fine — but if you accidentally do heavy computation in setup that should be in predict, you lose budget awareness.

Fix now: keep setup() lightweight. Move estimation logic to predict().

➡️ Next step

• Debugging Checklist
• FAQ
• Estimator Contract
• Scoring Model