Flopscope Primer

Flopscope is a numpy-compatible array library that tracks FLOPs analytically rather than timing them on hardware. Every arithmetic operation on a fnp.ndarray increments a FLOP counter instead of (or in addition to) performing the computation. This is how WhestBench enforces fair FLOP budgets across different machines.

Source: github.com/AIcrowd/flopscope

BudgetContext

All estimator predictions run inside a BudgetContext. When the budget is exhausted, a BudgetExhaustedError is raised and your predictions are zeroed out.

import flopscope as flops
import flopscope.numpy as fnp

with flops.BudgetContext(flop_budget=1_000_000) as ctx:
    x = fnp.ones((100, 100), dtype=fnp.float32)  # 10,000 FLOPs
    w = fnp.ones((100, 100), dtype=fnp.float32)  # 10,000 FLOPs
    y = x @ w  # 1,990,000 FLOPs: exceeds the 980,000 remaining

Raise the example budget to 10_000_000 to complete it; the total is 2,010,000 FLOPs.

You don't need to create BudgetContext yourself — the framework does it before calling your predict() method. The budget argument tells you how many FLOPs you have.

BudgetContext also supports wall_time_limit_s when you want a cooperative wall-clock limit in addition to the FLOP cap:

with flops.BudgetContext(flop_budget=1_000_000, wall_time_limit_s=2.0) as ctx:
    ...

The timer starts when the context is entered and is checked before and after each counted flopscope/NumPy call. If it is exceeded, flopscope raises TimeExhaustedError.

Operation FLOP Costs

Charges combine the analytical count, operation weight, dtype rate, and any complex factor. The table assumes float32 execution. Float64 work bills at 2× that rate; explicit dtype=/signature= and wider output destinations can increase it. Element-wise counts use the broadcast computation/output shape.

For example, a dense (100,100) @ (100,100) costs 1,990,000 FLOPs at float32; exp on 100 float32 elements costs 1,600. Reductions depend on the reduced axes and optional arguments; composite helpers include their internal work. Consult the full cost model and the actual ctx.summary() rather than treating an operation's weight as its whole price.

Billing and symmetry changes in 0.12.3

Use flopscope 0.12.3 when remeasuring Phase 2 candidates. It includes the 0.12.2 corrections and the 0.12.3 truncated-SVD model:

• Broadcast work: affected operations bill the shape they actually compute, including an enlarged out=/where= destination, var/std with an enlarging mean=, polynomial evaluation, and array tolerances in closeness tests. A small input does not imply a small bill when broadcasting expands the computation.
• Unary dtype selection: dtype=, signature=/sig=, and wider out= buffers can raise the bill above the input dtype's rate. For four float32 elements, ordinary exp(x) costs 64 FLOPs; exp(x, signature="d->d") costs 128.
• Composite operations: histogram edge construction and density normalization, per-axis histogram searches, length-one FFT stages, and polyfit covariance output now include work that was omitted. Follow the release notes for the affected forms rather than assigning a flat per-element price.
• Symmetry: einsum(..., symmetry=...) canonicalizes a validated result before attaching its tag; values accepted within tolerance can therefore change to make the declared symmetry exact. Positional output writes and shuffles invalidate stale tags, while scans and reordered outputs retain only the symmetry their axes support. Revalidate after modifying a tagged buffer.
• Truncated SVD: pricing includes rank-dependent quadratic and cubic work, capped at the corresponding full economy-decomposition price. Low-rank calls can retain a discount; near-full-rank calls can cost the full amount. Use flops.accounting.svd_cost(m, n, k=k, with_vectors=compute_uv) for the current float32 estimate, with matching full_matrices for a full decomposition. Truncated pricing is an analytical convention: the backend computes an exact economy SVD and slices the outputs, so a billing discount is not a runtime speed guarantee.

Inspect ctx.summary() and the installed flopscope version when comparing old and new profiles. The starter kit's mean- and covariance-propagation examples retain their documented FLOP totals under 0.12.3; that does not establish that another estimator's costs or predictions are unchanged.

Array Creation

import flopscope as flops
import flopscope.numpy as fnp

x = fnp.zeros(100)                          # 1D zeros
X = fnp.zeros((64, 100), dtype=fnp.float32)  # 2D zeros, explicit dtype
I = fnp.eye(100, dtype=fnp.float32)          # identity matrix
a = fnp.array([1.0, 2.0, 3.0])             # from list
b = fnp.asarray(numpy_array)                # convert from numpy (free)

Only zero/empty allocation and no-copy conversions are free. Nonzero fills, identity diagonals, materializing copies, and dtype conversions are billed.

Random Number Generation

import flopscope as flops
import flopscope.numpy as fnp

rng = fnp.random.default_rng(42)            # seeded RNG
x = rng.standard_normal((1000, 64))        # Gaussian samples
x = x.astype(fnp.float32)                   # billed cast; request float32 draws instead

Constructing a generator is free; generating samples is billed. Prefer rng.standard_normal((1000, 64), dtype=fnp.float32) when float32 is sufficient, rather than paying for float64 draws and a subsequent cast.

Budget Inspection

Use budget.summary() for the current explicit context and flops.budget_summary() for the accumulated session/global view:

with flops.BudgetContext(flop_budget=10_000_000) as ctx:
    # ... your computations ...
    print(ctx.summary())         # current context only
    flops.budget_summary()         # prints the process/session-wide summary
    print(ctx.flops_used)       # integer FLOP count

Both summaries also include four timing fields that satisfy a strict decomposition identity, wall_time_s = flopscope_backend_time_s + flopscope_overhead_time_s + residual_wall_time_s:

• wall_time_s: total elapsed time in the context
• flopscope_backend_time_s: time spent inside counted flopscope numpy kernels
• flopscope_overhead_time_s: time spent inside flopscope's own dispatch (wrapper preambles, FLOP bookkeeping, namespace push/pop)
• residual_wall_time_s: everything else - participant Python, GC, and any ops not attributed to a flopscope backend or callback bucket (see the 0.7.0 note below)

This decomposition lets you see whether time is going to numpy compute, framework dispatch, or your own Python.

flopscope 0.7.0 timing re-attribution. As of flopscope 0.7.0, data-movement NumPy ops (concatenate, stack-family, tile, repeat, take, pad, …) are timed as flopscope_backend_time_s, not residual_wall_time_s; Python-callback ops bill their callback time to residual. The identity wall_time_s = flopscope_backend_time_s + flopscope_overhead_time_s + residual_wall_time_s still holds; FLOP counts are unchanged.

WhestBench-specific limits

Flopscope's BudgetContext measures wall_time_s, flopscope_backend_time_s, flopscope_overhead_time_s, and residual_wall_time_s. It also accepts wall_time_limit_s, which it checks while counted flopscope operations run.

WhestBench exposes some of those concepts as run-level CLI knobs:

• --wall-time-limit: passed through to the estimator's BudgetContext
• --residual-wall-time-limit: enforced by WhestBench after predict() returns, using the reported residual_wall_time_s. Because residual_wall_time_s no longer includes flopscope's own dispatch time, this gate measures only your Python work — not the framework's bookkeeping tax.

So if you see time_exhausted, that came from Flopscope's wall_time_limit_s. If you see residual_wall_time_exhausted, that came from WhestBench scoring logic comparing Flopscope's measured residual_wall_time_s with the configured --residual-wall-time-limit.

Residual wall time: gated, not priced

Residual wall time is the part of predict() that flopscope does not meter. There are two ways to stop that being a free lunch, and WhestBench supports both — the difference is one number, λ:

The default is gated, so C_m = F_m — the FLOP budget means exactly what it says, and the two resources do not trade against each other. This is the current round's design. The priced regime is the earlier one, at λ = 1e11 FLOP-equivalents per second.

The general formula covers both, since λ = 0 collapses it:

C_m = F_m + λ · R_m

• F_m = analytical FLOPs counted by flopscope (flops_used)
• R_m = residual wall time — the third bucket of the time decomposition. Specifically, residual_wall_time_s = wall_time_s − flopscope_backend_time_s − flopscope_overhead_time_s. This is participant Python (loops, control flow), GC pauses, and uninstrumented numpy. It explicitly excludes flopscope's own dispatch overhead (the second bucket).
• λ = the residual price, in FLOP-equivalents per second. Defaults to 0 (whestbench.budget.DEFAULT_LAMBDA_FLOPS_PER_SECOND); set per-run with whest run --lambda-flops-per-second.

The combined C_m is capped at B_m = flop_budget. If C_m > B_m, the MLP is marked combined_budget_exhausted and the prediction is replaced with zeros. Under the default gated regime C_m = F_m, so that post-hoc check is a backstop on the same FLOP count that budget_exhausted already guards — blowing the residual cap surfaces separately, as residual_wall_time_exhausted.

Reproducing an earlier round

Nothing here is hard-coded to a round. To re-score against the priced regime, ask for its rate and turn the gate off — it did not exist then, and leaving it armed would fail MLPs against a rule they were never scored under:

whest run --estimator estimator.py \
  --lambda-flops-per-second 1e11 \
  --no-residual-wall-time-limit \
  --flop-budget 272000000000 \
  --wall-time-limit 60

All four matter. --wall-time-limit is easy to forget because it is not part of the pricing change, but Phase 1 graded predict() at 60 s and the current default is 120 s: a submission whose predict() took between 60 s and 120 s was time_exhausted in the graded round — zeroed, no compute discount — and would otherwise score normally here, silently and in the participant's favour. Pass --dataset with that round's revision too, so the MLP shape and seeds match; the flags above only restore the scoring rules.

whestbench.budget.PHASE1_LAMBDA_FLOPS_PER_SECOND is that rate, kept as a named constant for the same purpose. run_config.lambda_flops_per_second and run_config.residual_wall_time_limit_s in every report record which regime produced it.

Common Gotchas

numpy arrays still count FLOPs. Since fnp.ndarray is backed by numpy, a raw numpy array passed to flopscope operations will still be tracked. Convert explicitly with fnp.array() or fnp.asarray() — but those conversions are themselves subject to the numeric-dtype rule below, so fnp.array([1.0, None]) and fnp.array(['a', 'b']) raise UnsupportedDtypeError rather than producing an object or string array. Build numeric data with fnp.array(..., dtype=fnp.float64) from values that are already numeric scalars, and keep ragged or mixed data in a Python list of numeric arrays instead of one array. Converting with plain numpy first is not a remedy: the grader sandbox ships no numpy.

Pythonic operators are tracked. x @ w counts the same FLOPs as fnp.matmul(x, w). Use whichever reads better.

dtype decides both cost and admission. dtype scales FLOP cost: the charged cost is flop_cost * dtype_rate * complex_factor * weight, and float64 carries a rate of 2.0, so a float64 operation costs twice the same operation in float32. dtype also decides admission — flopscope accepts only numeric dtypes (dtype.kind in "biufc": bool, signed and unsigned integer, float, complex). Anything else raises UnsupportedDtypeError (importable from flopscope.errors) wherever it reaches a registered operation, whether as an operand, an explicit dtype=, a fill value or distribution parameter, or an out= destination. The one carve-out is a dtype NumPy materialises with zero itemsize, such as an empty structured spec ('V0'); 'U0' and 'S0' are not exempt, because NumPy promotes them to 'U1'/'S1' on allocation.

Testing

Use flopscope's testing utilities:

import flopscope as flops
import flopscope.numpy as fnp

fnp.testing.assert_allclose(actual, expected, atol=1e-6)
fnp.testing.assert_array_equal(actual, expected)

These work like numpy's testing functions but on flopscope arrays.