Sourced from whest-starterkit @ 96e308c.

Profile simulation performance

← Documentation

🎯 When to use this page

The starter kit uses flopscope as its single simulation backend. The profile-simulation command lets you verify the backend computes correct values and see how its wall-clock cost scales with network size.

The whest profile-simulation command benchmarks the backend's correctness and timing. To manage your estimator's FLOP budget during development, see Manage Your FLOP Budget.

Use this page when you want to:

• Verify flopscope is installed and correct — the profiler runs a correctness check before timing anything.
• Triage performance problems (for example, "fast on my laptop, slow in CI") — compare median wall-clock time for run_mlp and sample_layer_statistics across widths, depths, and sample counts.
• Collect reproducible timing data — JSON output includes correctness results and per-configuration timing (median time, raw per-iteration times, warmup time) across network sizes, plus hardware/library-version metadata.

🚀 Do this now

1. Run a quick profile

uv run whest profile-simulation --preset quick

This finishes in seconds and gives you a first look at correctness and timing across a few network sizes.

2. Run the standard profile

uv run whest profile-simulation

The default standard preset tests two widths (64, 256) and three depths (4, 32, 128). It shows how run time scales across network sizes.

3. Save results for comparison

uv run whest profile-simulation --output results.json

The JSON file contains correctness results and per-configuration timing data (median time, raw per-iteration times, warmup time) across all tested configurations, plus hardware and library-version metadata.

Choosing presets

Use quick for a fast sanity check and standard for development decisions.

Understanding the output

The terminal table shows:

• Hardware — platform, CPU, RAM, and Python/NumPy versions, for reproducibility across machines.
• Correctness — PASS or FAIL for the flopscope backend. A FAIL indicates a version mismatch or installation problem.
• Detail — one row per (backend, width, depth, n_samples) combination, with median wall-clock time for run_mlp and sample_layer_statistics.

Add --verbose for the full Timing Results table: one row per (backend, operation, width, depth, n_samples) combination, with columns for median time, speedup vs NumPy, and status. It renders only in rich output, so add --format rich when piping or redirecting: uv run whest profile-simulation --verbose --format rich | tee profile.txt. For machine-readable data prefer --output results.json; the JSON carries the full per-configuration detail regardless of --verbose.

Common workflows

Debug a correctness failure

If the correctness check shows FAIL:

uv run whest profile-simulation --preset quick --debug

The error message indicates whether the issue is a numerical tolerance failure or a missing dependency.

Export timing data across machines or code changes

uv run whest profile-simulation --preset exhaustive --output timing_data.json

To see whether a code change, dependency bump, or a different machine affected simulation performance, compare the timing array's median_time field between two JSON files.

This command reports wall-clock timing only, not FLOP counts. To see how many FLOPs your estimator consumes, wrap it in a flopscope.BudgetContext and call ctx.summary(); see Manage Your FLOP Budget.

✅ Expected outcome

• The terminal displays a formatted table with correctness and timing results.
• If you pass --output, the command writes a JSON file with hardware info, correctness results, and per-configuration timing data.

➡️ Next step

• CLI Reference — points to the upstream whest CLI reference for the full flag list
• Use Evaluation Datasets — pre-create datasets for faster iteration
• Validate, Run, and Package — score your estimator