combine_split_datasets

combine_split_datasets(input_dirs: "'list[Path | str]'", *, output_dir: "'Path | str'", default_split: "'str | None'" = None, write_prepared_arrow: 'bool' = True) -> 'Path'

Combine N complete single-split datasets into a multi-split dataset directory.

Each input must be a complete (non-partial) single-split schema-3.0 dataset
directory. Inputs must agree on schema_version, format, backend, seed_protocol,
width, depth, and n_samples. Split names (inferred from each input's single
parquet filename) must be pairwise distinct.

Output is a multi-split dataset directory with one parquet file per input
split, a multi-split metadata.json (``splits:`` dict), and a rendered README.

Args:
    input_dirs: Paths to complete single-split dataset directories.
    output_dir: Destination path; must not exist.
    default_split: Optional name of the split that downstream consumers
        should fall back to when a multi-split dataset is given without
        an explicit ``--split``. Must be one of the present split names;
        otherwise raises ``MergeIncompatibleError``. Recorded as
        ``default_split`` at the top level of the multi-split
        ``metadata.json``. ``whest run`` honours it when ``--split`` is
        omitted on a multi-split dataset.
    write_prepared_arrow: When True (default), also emits a
        ``prepared/<split>/`` directory next to ``data/`` containing the
        `Dataset.save_to_disk()` output for each split. The
        ``prepared_splits`` block is added to ``metadata.json`` so
        consumers (``whestbench.load_dataset``, ``whest run``) prefer
        this pre-arrow'd path on HF Hub and skip the parquet→arrow
        conversion entirely on cold cache. Set False to skip when the
        prepare cost matters more than the runtime win (e.g. tests).

Returns:
    Path to the output directory.

Raises:
    MergeIncompatibleError: inputs disagree on invariants, contain a partial,
        have duplicate split names, ``default_split`` is unknown, or input
        list is empty.
    FileExistsError: output_dir already exists.