Sourced from whest-starterkit @ 96e308c.

Manage your FLOP budget

← Documentation

🎯 When to use this page

Use this page to understand how FLOP budgets work and how to optimize your estimator to stay within budget.

Why FLOPs, not wall-clock time

This challenge scores estimators by analytical FLOP count, not execution time. Every mathematical operation your estimator performs is tracked by flopscope, a NumPy-compatible library that counts floating-point operations deterministically from tensor shapes and dtypes.

This means your FLOP count is hardware-independent: the same estimator produces the same F_m on a laptop and on the grader. Your effective compute is C_m = F_m. Nothing else contributes to your score, so you cannot trade wall-clock time for FLOPs or FLOPs for wall-clock time.

Wall time is a limit, not part of the score. Residual wall time (anything flopscope does not meter) is capped at 400 ms per MLP, and predict() as a whole is capped at 120 s per MLP. If you cross either limit, the harness replaces that MLP's prediction with zeros. Residual time exists for plumbing (unpacking mlp, control flow around your fnp calls, assembling the returned array), not for computation: doing real work outside flopscope.numpy is prohibited, not merely expensive. See Performance Tips and Is scoring hardware-dependent?.

For the full flopscope API and cost model, see the flopscope documentation.

Which operations cost FLOPs

Two consequences:

• fnp.matmul on (n, n) matrices costs O(n^3) FLOPs; a matrix-vector product costs O(n^2). The cost is cubic in the width, so at the Phase 2 width of 1024 matrix work dominates everything else. Even in an estimator that only ever does matrix-vector products, matmul is 77.42% of the total (see the worked walkthrough below).
• Every cost above is also scaled by a dtype_rate: 1.0× for 32-bit-or-smaller dtypes, 2.0× for float64/int64 (up to 4.0× for float128). The executed loop determines the billing dtype: operand promotion, explicit unary dtype=/signature=/sig=, and wider out= destinations can all affect the charge. A float64 operand often raises the rate, but an explicitly selected float32 loop can differ from that default. See the flopscope primer for the full weight/dtype-rate table.

Check your budget usage

The Phase 2 per-MLP budget is B_m = 2**41 = 2,199,023,255,552 FLOPs. To see how many FLOPs your estimator consumes, wrap its logic in a BudgetContext with that budget:

import flopscope as flops

with flops.BudgetContext(flop_budget=2_199_023_255_552) as budget:
    result = estimator.predict(mlp, budget=2_199_023_255_552)

print(f"FLOPs used: {budget.flops_used:,}")
print(f"FLOPs remaining: {budget.flops_remaining:,}")

If you also want a wall-clock limit while debugging locally, set wall_time_limit_s on the same BudgetContext:

with flops.BudgetContext(
    flop_budget=2_199_023_255_552,
    wall_time_limit_s=2.0,
) as budget:
    result = estimator.predict(mlp, budget=2_199_023_255_552)

Get a per-operation breakdown

To see which operations consume the most FLOPs, use budget.summary() for the current explicit context or flops.budget_summary() for the session/global view:

import flopscope as flops

with flops.BudgetContext(flop_budget=2_199_023_255_552) as budget:
    result = estimator.predict(mlp, budget=2_199_023_255_552)
    print(budget.summary())

flops.budget_summary()

This prints a table showing each operation's name, call count, and cumulative FLOP cost.

The same summaries also show timing data:

• wall_time_s: total elapsed time for the context
• flopscope_backend_time_s: time spent inside counted flopscope backend calls
• flopscope_overhead_time_s: time spent inside flopscope dispatch and bookkeeping
• residual_wall_time_s: participant Python (loops, control flow), GC, and Python-callback op time; as of flopscope 0.7.0, data-movement NumPy ops (concatenate, stack, tile, repeat, take, pad, …) count as flopscope_backend_time_s, not residual

In whest run, the CLI flags map to these concepts as follows:

• --wall-time-limit: forwards a wall-clock limit into the estimator's BudgetContext. Default 120.0, already the graded cap.
• --residual-wall-time-limit: adds a WhestBench scoring check on the reported residual_wall_time_s. Default 0.4, already the graded cap, so a plain whest run already gates you exactly as the live round does. Pass either flag only to make the local run stricter than the grader.
• --setup-timeout: wall-clock limit on the one-time setup() call. Default 5.0, the graded cap. Exceeding it fails the whole submission, not one MLP.

Interpret whest run output

When you run your estimator with whest run, the per-MLP report includes:

• flops_used: total FLOPs your estimator consumed for that MLP.
• budget_exhausted: true if your estimator exceeded the FLOP budget; predictions were zeroed.
• residual_wall_time_s / residual_wall_time_exhausted: time spent outside metered flopscope ops, and whether it crossed the limit. On the grader the limit is 400 ms per MLP and crossing it zeroes that MLP's predictions.
• final_layer_mse / all_layers_mse: your prediction accuracy (lower is better).

If budget_exhausted is true, the harness discarded your predictions and you need to reduce FLOP usage. If residual_wall_time_exhausted is true, the problem is not FLOPs at all. See Performance Tips.

Worked walkthrough: mean propagation, line by line

The table below profiles examples/02_mean_propagation.py on the competition shape (width=1024, depth=16). Numbers are aggregated across all 16 layers; per-layer cost is roughly the row total divided by 16. To reproduce these numbers, call ctx.summary() inside a flopscope.BudgetContext after a single predict() call (profiled under flopscope 0.12.3; exact totals checked by tests/test_flopscope_cost_docs.py).

The full ~86.6 M FLOPs spends only ~0.004% of the 2,199,023,255,552-FLOP grader budget, so mean propagation stays well below the multiplier floor at this shape. See Scoring Model.

Three takeaways:

• matmul dominates. ~77% of predict() cost is the two matmuls per layer (the pointwise ReLU-moment terms, multiply, are the visible ~20% remainder). Halving the matmul count (for example, switching to a diagonal-only formulation, or fusing into a single einsum like examples/03_covariance_propagation.py does for the symmetric cov-update) recovers most of that.
• dtype is worth more than any single op here. This walkthrough profiles the example as shipped, which seeds mu/var at dtype=fnp.float32. Leave that off and everything the two arrays touch (the matmuls included) bills at the float64 2× rate. Dropping the dtype= seeds alone costs 153,978,880; dropping them and the two .astype(fnp.float32) recasts (the fully-float64 estimator) costs 153,913,344, which is lower only because it no longer pays the 65,536 FLOPs those 32 casts cost, and 77.6% more than as shipped, that is, the float32 seeding is a 43.7% saving, for a final-layer MSE identical to five significant figures. mlp.weights is already float32, so the float64 buys nothing. See Stay in float32.
• Sqrt, divides, and clamps stay cheap. Don't restructure your code to avoid them: sqrt, true_divide, and maximum each cost 1,024 FLOPs per call here (1024 elements at the float32 rate), trivial next to the matmuls.

The same pattern holds for examples/03_covariance_propagation.py, where the O(width³) symmetry-aware einsum, plus the per-layer as_symmetric() re-validation that keeps its symmetric rate under flopscope ≥ 0.10.0, lands at ~51.7 B FLOPs per predict() (51,709,240,799 exactly; 2.351% of the grader budget, profiled the same way under flopscope 0.12.3). That is 597× more expensive than mean propagation, because a full covariance requires more arithmetic than a diagonal variance. It seeds at float32 too; leaving the default would put it at 103,407,495,614, a 99.98% increase, not quite a doubling, because the 32 stats.norm calls bill float64 either way.

Optimization tips

1. Matmul dominates. Each fnp.matmul(W.T, mu) on a (width, width) matrix costs O(width^2) FLOPs per layer. Reducing the number of matmuls (or their dimensions) has the biggest impact.

2. Diagonal approximations save FLOPs. Mean propagation uses diagonal variance (O(width^2) per layer) instead of full covariance propagation (O(width^3) per layer): 86,639,616 FLOPs against 51,709,240,799 at this shape. Both sit under the score's multiplier floor of max(0.1, C_m / B_m): below 10% of the budget the multiplier does not move, so extra compute that improves accuracy does not change your score up to that point. Choose the level of approximation that maximises accuracy, not the cheapest one.

3. Only fnp.zeros()/fnp.empty() and no-copy views are free. Since flopscope 0.9, fnp.array(), fnp.ones(), and fnp.eye() each bill 1 FLOP per element at float32 (2× that at the float64 default). eye bills only its diagonal length, not the full matrix. They're still cheap next to any matmul, but calling them inside a hot loop is no longer free, so prefer fnp.zeros() or a no-copy fnp.asarray() for placeholders you'll overwrite anyway.

4. Pick one strategy per estimator. Use either mean propagation or full covariance as your default implementation, then optimize it for the fixed 2**41 budget.

5. Don't try to move work out of the FLOP budget. Computation performed outside flopscope.numpy (bundled numpy or BLAS, compiled kernels, FFI, threads or subprocesses) is prohibited and grounds for disqualification, not a cheaper way to gain accuracy. See Performance Tips.

If you believe flopscope is mispricing an operation, report it at arc-whestbench@aicrowd.com rather than building around it.

➡️ Next step

• Write an Estimator
• Scoring Model
• Profile Simulation
• Estimator Contract