Sourced from whest-starterkit @ aaa3882.

Profile Simulation Performance

← Documentation

🎯 When to use this page

The starter kit uses flopscope as its single simulation backend — analytical FLOP counting replaces wall-clock timing across multiple backends. The profile-simulation command lets you verify FLOP accounting correctness and explore how FLOP costs scale with network size.

This page covers the whest profile-simulation command for benchmarking the backend's correctness and FLOP scaling. For managing your estimator's FLOP budget during development, see Manage Your FLOP Budget.

Use this page when you want to:

• Verify flopscope is installed and correct — the profiler runs a pre-flight correctness check before reporting FLOP data.
• Understand FLOP scaling — see how FLOP costs grow with width, depth, and budget so you can calibrate your estimator's budget usage.
• Collect reproducible profiling data — JSON output includes correctness results and FLOP accounting across network sizes.

🚀 Do this now

1. Run a quick profile

whest profile-simulation --preset quick

This finishes in seconds and gives you a first look at FLOP accounting correctness and scaling.

2. Run the standard profile

whest profile-simulation

The default standard preset tests two widths (64, 256) and three depths (4, 32, 128). It gives a reliable picture of FLOP cost scaling across network sizes.

3. Save results for comparison

whest profile-simulation --output results.json

The JSON file contains correctness results and FLOP accounting data across all tested configurations.

Choosing presets

Use quick for a fast sanity check and standard for development decisions.

Understanding the output

The terminal table shows:

• Pre-flight Correctness Check — PASS or FAIL for the flopscope backend. A FAIL indicates a version mismatch or installation problem.
• FLOP Accounting Results — one row per (width, depth) combination showing the total FLOPs reported by flopscope for a forward pass.

Common workflows

Debug a correctness failure

If the correctness check shows FAIL:

whest profile-simulation --preset quick --debug

The error message will indicate whether the issue is a numerical tolerance failure or a missing dependency.

Export FLOP data for estimator calibration

whest profile-simulation --preset exhaustive --output flop_data.json

Use the FLOP counts in the JSON to understand how much budget your estimator consumes per layer, and tune your budget-allocation logic accordingly.

✅ Expected outcome

• The terminal displays a formatted table with correctness and FLOP accounting results.
• If --output is provided, a JSON file is written with full profiling data.

➡️ Next step

• CLI Reference: profile-simulation — full flag reference
• Use Evaluation Datasets — pre-create datasets for faster iteration
• Validate, Run, and Package — score your estimator