Release process

This document is the authoritative reference for cutting a new release of whestbench to PyPI. It covers the steady-state flow, the one-time setup that must happen outside the repo, and a few troubleshooting notes.

TL;DR (steady-state)

git checkout main && git pull origin main
uv run cz bump --dry-run                  # preview the next version + CHANGELOG entry
uv run cz bump --files-only --changelog   # write pyproject version + CHANGELOG.md — no commit or tag yet
uv lock                                   # uv.lock records whestbench's own version; stale after every bump
git add pyproject.toml CHANGELOG.md uv.lock
git commit -m "bump: version <old> → <new>"
git tag -a v<new> -m "v<new>"             # annotated, like every existing release tag
git push --follow-tags                    # tag push triggers the publish workflow
# … the publish workflow builds, publishes to PyPI, and creates the
# GitHub Release. NOTE: with no Required Reviewers on the `pypi`
# environment (the current state — see "One-time setup" §2) it runs
# straight through with NO approval pause.

Why not plain uv run cz bump? It commits and tags in one shot, before uv.lock can be refreshed — so the bump commit ships a stale lockfile and the next uv run on anyone's machine rewrites it as uncommitted drift. Every real bump commit (v0.13.0's 68b50bb, v0.14.0's a2d65e3) includes the relocked file. If you ran plain cz bump anyway: uv lock, git add uv.lock && git commit --amend --no-edit, and re-point the tag with git tag -fa v<new> -m "v<new>" before pushing.

Pre-release tags follow the same files-only flow with --prerelease alpha added (uv run cz bump --files-only --changelog --prerelease alpha); the relock, commit, and tag steps are identical, with the prerelease version to tag (e.g. v0.5.0a0) shown in the bump output.

What happens after git push --follow-tags

The tag push fires .github/workflows/pypi-publish.yml, which:

1. Builds the sdist + wheel with uv build.
2. Pauses for approval in the pypi GitHub environment — only if Required Reviewers are configured on it. As of v0.14.0 they are not (see "One-time setup" §2), so the publish proceeds immediately.
3. Publishes to PyPI via Trusted Publishing (OIDC; no API token stored in repo secrets).
4. Creates a GitHub Release whose body is the matching CHANGELOG section for the tag.

End result: uv add whestbench / pip install whestbench works ~2 minutes after the tag push (or after the approval click, once Required Reviewers are configured).

One-time setup (per maintainer, per repo)

Before the first release will succeed, two things must be configured outside the repo.

1. PyPI Trusted Publisher

On pypi.org, as an account with Owner or Maintainer rights on the whestbench project (or as the user creating it, if not yet published):

1. "Your projects" → whestbench → "Publishing" → "Add a pending publisher" (or "Add a publisher" if the project already exists).
2. Fill in:
   • PyPI project name: whestbench
   • Owner: AIcrowd
   • Repository name: whestbench
   • Workflow filename: pypi-publish.yml
   • Environment name: pypi

PyPI's "pending publisher" feature allows trusted publishing to succeed on the very first publish of a brand-new project name.

2. GitHub pypi environment

In the AIcrowd/whestbench repo on GitHub:

1. Settings → Environments → "New environment" → name: pypi.
2. Enable "Required reviewers".
3. Add yourself (and any other release maintainers) as reviewers.
4. Save.

Without this, publishes proceed without a human approval gate. The Trusted Publishing OIDC handshake will still work — there is just no gate to abort a bad tag.

Current state (as of v0.14.0, 2026-07-31): the pypi environment on AIcrowd/whestbench has no Required Reviewers — the v0.14.0 publish ran straight through with no approval pause (workflow run 30600703882). Until a repo admin adds Required Reviewers under Settings → Environments → pypi, treat every v* tag push as an immediate, ungated publish: everything must be verified green before git push --follow-tags.

How CHANGELOG entries get into the GitHub Release

The publish workflow extracts the body of the matching ## v<version> section in CHANGELOG.md using an awk script and uses it as the GitHub Release notes. Commitizen writes section headers in the ## v<version> (<date>) form, which the workflow expects.

When promoting an existing ## Unreleased section to a versioned release manually (rather than via cz bump), use the same header format: ## v0.4.0 (2026-05-26).

If no matching section is found, the workflow falls back to a default body: Release v<x.y.z>\n\nSee CHANGELOG.md for details.

Troubleshooting

Publish job fails with "Trusted publisher not configured"

PyPI side is not configured. Re-check step 1 of "One-time setup". The workflow filename and environment name must match exactly (pypi-publish.yml, pypi).

Publish job fails with "File already exists on PyPI"

A version was previously uploaded and yanked. PyPI does not allow re-uploading the same version, even after a yank. Resolution: delete the tag locally and on the remote, bump to the next version, retag:

git tag -d v0.5.0
git push origin :refs/tags/v0.5.0
# then run the TL;DR flow from `cz bump --dry-run` onward — it lands on
# the next patch version (v0.5.1) and retags

GitHub Release step fails after PyPI succeeded

The package is on PyPI; only the GitHub Release is missing. Re-run the workflow on the same tag from the GitHub Actions UI. The github-release job's gh release create is the only remaining side effect and is idempotent against the existing tag (will fail if a release already exists, succeed if not).

cz bump --dry-run previews an unexpected version

The previewed version is computed from conventional-commits types in the commit range since the last tag. feat → minor bump (under v1.x behaviour: still minor while major_version_zero = true in [tool.commitizen]), fix → patch, feat! or BREAKING CHANGE → minor while major_version_zero = true, else major. To bump to a specific version explicitly, use cz bump --increment PATCH|MINOR|MAJOR.

Pin updates for flopscope

Whestbench pins flopscope>=0.11.0,<0.12.0 and flopscope-server>=0.11.0,<0.12.0. When flopscope ships a new minor or major version, bump these floors in pyproject.toml and re-run uv lock before cutting the next whestbench release. (Out of scope for an automated workflow; flag if Dependabot becomes worth the noise.)

After any flopscope bump, re-run tests/test_torch_flop_synthesis.py first. It asserts the torch bake's closed-form FLOP count equals flopscope's actual count exactly, so it is the gate on whether a bump invalidates already-published dataset revisions. A bump that moves it means baked sampling_budget_breakdown figures no longer describe the accounting in force — under an immutable HF tag, where nothing will fail loudly.

If the harness ever needs to track flopscope's main between releases, do it with a [tool.uv.sources] redirect rather than a direct URL in [project.dependencies] — PyPI rejects direct-URL dependencies in an uploaded distribution, and uv strips tool.uv.sources from built wheel metadata. Two consequences while such a redirect is in place: do not cut a release (the wheel would declare the plain range and resolve a different flopscope than the one tested), and expect CI — which syncs with --upgrade-package flopscope --upgrade-package flopscope-server — to go red on upstream commits that no whestbench change caused.

A flopscope bump that changes absolute FLOP counts requires a re-baseline + full re-eval plan (see the 0.5.0 incident); 0.11.0 is one — a contraction whose combined operand rank exceeds the 52-letter subscript budget now bills the honest FMA count instead of multiplies only, so tensordot calls built on singleton-axis padding bill ~2x, and fnp.ix_ moving from weight 0.0 to 1.0 is a second, smaller mover. A bump that only re-attributes timing (e.g. 0.7.0: data-movement → backend) shifts residual-based scores and warrants a measure-then-decide re-eval. A bump that only raises the local in-process estimate to match what the grader was already charging is participant-visible but triggers no re-evaluation: 0.11.0's linalg, cross, outer, contraction, fft and bmat return-type wrapping is that case — grader-side counts do not move.

Docs

Docs deploy automatically on push to main (.github/workflows/docs.yml). To adopt new starter-kit curriculum, run python scripts/bump_starterkit_pin.py (it bumps the pin to the latest starter-kit main and lists the changed docs); then make docs-build and commit. The starter-kit drift workflow opens an issue when the pin falls behind main.