Skip to content

Latest commit

 

History

History
241 lines (195 loc) · 11.7 KB

preprocessing_3d.md

File metadata and controls

241 lines (195 loc) · 11.7 KB

Preprocessing 3D Datasets

This guide turns raw 3D volumes (NIfTI or anything else SimpleITK reads) into the generic_3d_dataset format that glovita trains on. Everything generic lives in generic_3d.py, which has two stages:

  1. preprocess reorients and normalises the raw volumes, resamples them to one or more target spacings, and writes the labels and the splits.
  2. derive (optional) takes a preprocessed dataset and either downsamples whole volumes to a fixed shape or crops a fixed-size region of interest from them.

Each dataset gets one adapter file next to it that describes its cases and its specific settings and calls both stages. The PANORAMA one is panorama.py.

All scripts in that folder run by name with glovita_preprocess:

glovita_preprocess                                  # list the scripts
glovita_preprocess panorama preprocess --help       # options of one script and stage
glovita_preprocess panorama preprocess --raw_dir RAW --out_dir OUT
glovita_preprocess generic_3d derive resize --src OUT/MyData_1mm --dst OUT/MyData_1mm_downsampled192

A new adapter file in the folder is picked up automatically. It only needs a main() that reads the command line.

Install the extra dependencies first:

pip install -e .[preprocessing]   # SimpleITK, nnU-Net (resampling), psutil

Output Format

Both stages write a dataset that --data.dataset generic_3d_dataset loads directly (see nnfoundation.md):

<dataset>/
├── images/<case>.b2nd   float32 (C, z, y, x), oriented and normalised
├── masks/<case>.b2nd    uint8 (1, z, y, x) on the image grid (only for cases with a mask)
├── dataset.json         num_classes, subtask, spacing, normalisation, ... (see below)
├── labels.json          {case: class index} or {case: multi-hot list}
├── splits.json          [{"train": [...], "val": [...], "test": [...]}, ...] (one entry per fold)
└── cases.csv            per-case geometry and QC

The volumes are blosc2 files. Each file's metadata holds case_id, spacing (the output spacing, z/y/x) and orig_spacing.

dataset.json always contains num_classes and subtask, which the data config reads. It also contains spacing, a list of three numbers in array axis order. The 3D augmentation reads it to restrict rotations and scaling to the in-plane axes when the data is anisotropic (see augmentation/policies.md). Derived datasets whose spacing differs from case to case store median_spacing instead.

Stage 1: preprocess

What it does to each case

  1. Read: it reads the image channels and the mask. If the mask lies on a different grid (size, spacing, origin or direction), it is resampled onto the image grid with nearest-neighbour interpolation. Extra image channels are resampled onto the first channel's grid.

  2. Reorient to --orientation (default LPS, or none to keep the stored orientation). nnssl and nnU-Net-style pipelines ignore direction cosines, so datasets with mixed orientations have to be unified first.

  3. Normalise each channel before resampling, as nnssl does:

    • ct: clip below --clip_min (e.g. -1024 HU), then z-score over the whole volume
    • zscore: z-score over the whole volume
    • none: keep the values

    Give one mode for all channels, or one mode per channel (e.g. --intensity ct zscore).

  4. Resample with nnU-Net (resample_data_or_seg_to_shape) to every target spacing. Images use order 3, masks order 1, and both use order 0 along z when the data is anisotropic. Targets:

    • 1mm: isotropic 1 mm
    • median: the per-axis median of the oriented spacings of all cases in the table
    • ZxYxX: an explicit spacing, e.g. 1.5x0.75x0.75

    Each target becomes its own dataset, <out_dir>/<name_prefix>_<target>.

Splits

The script first holds out a fixed test set of --test_frac, then makes --n_folds folds of the remaining cases. Both are stratified by the label and grouped by --group_column (e.g. the patient), so no group is split across train, val and test. Every fold carries the same test list. Multilabel targets are stratified by their label combination. --n_folds 0 writes no splits.json.

Resources and resuming

  • Memory: each case's peak memory is estimated from its header, and cases only start while the estimates fit --mem_budget_gb and --min_free_gb stays free. If free memory ever falls below --abort_free_gb, all workers are killed. The workers run at nice 19 with idle I/O priority.
  • Resuming: outputs are written atomically and every case is logged to <out_dir>/<name_prefix>_preprocess_log_<targets>.jsonl. Rerunning the same command skips finished cases and retries failed ones.
  • --metadata_only rewrites dataset.json, labels.json, splits.json and cases.csv from the log without processing any volume. Use it, for example, after changing the split settings.
  • --case_ids a b c processes only these cases. The median spacing and the splits are still computed from all cases.

Running it from a case table (no adapter)

The CLI reads a CSV with one row per case:

column meaning
case_id unique id, also the output file name
image_path the image; several channels separated by ;
mask_path optional segmentation (leave empty for cases without one)
label column (--label_column, default label) class index, or a multi-hot list as JSON, e.g. [0, 1, 1]
group column (--group_column) optional, e.g. the patient id
other columns copied into cases.csv when listed in --case_columns
glovita_preprocess generic_3d preprocess \
  --cases cases.csv --out_dir /data/MyData --name_prefix MyData \
  --spacings 1mm median --intensity ct --clip_min -1024 \
  --label_column label --group_column patient_id --case_columns patient_id scanner \
  --num_workers 4 --mem_budget_gb 20

This writes /data/MyData/MyData_1mm and /data/MyData/MyData_median. Run glovita_preprocess generic_3d preprocess --help for all options.

Writing a dataset adapter

Keep one adapter file per dataset next to generic_3d.py. It only describes what is specific to the dataset: where the files are, which cases to use, the labels and groups, the intensity handling, any extra dataset.json entries, and the parameters of the derived datasets. Everything else comes from generic_3d:

import pandas as pd
from generic_3d import DeriveConfig, Preprocess3dConfig, add_run_arguments, derive_kwargs, preprocess, run_kwargs


def load_cases(raw_dir) -> pd.DataFrame:
    # one row per case: case_id, image_path, mask_path, the label and group columns, extra columns
    ...


def config(args) -> Preprocess3dConfig:
    return Preprocess3dConfig(
        name_prefix="MyData",
        intensity=["ct"], clip_min=-1024.0,       # or ["zscore"], one entry per channel, ...
        label_column="label", group_column="patient_id",
        case_columns=["patient_id", "scanner"],
        mask_labels={"background": 0, "lesion": 1}, max_mask_label=1,
        class_names=["negative", "positive"],
        # optional: add, rename or reorder dataset.json entries
        dataset_json_fn=lambda base, name, target, labels: {**base, "modality": "CT"},
        **run_kwargs(args),                       # --out_dir, --spacings, --num_workers, ...
    )


def derive_config(args) -> DeriveConfig:
    return DeriveConfig(
        roi_labels=[1, 2],                        # roi_crop: labels whose bounding box centres the crop
        lesion_label=1,                           # optional QC: coverage of this label by the crop
        case_columns=["patient_id"],              # carried over from the source cases.csv
        **derive_kwargs(args),                    # mode, --src, --dst, --target_shape, ...
    )

The PANORAMA adapter is a complete example. Its main() adds the shared options with add_run_arguments / add_derive_arguments, and it has an extra extract stage that unzips the downloaded images.

Stage 2: derive (fixed-size datasets)

This stage starts from a preprocessed dataset (images/, masks/, labels.json, splits.json, dataset.json) and writes a new dataset with one fixed shape per case (--target_shape, default 192 192 192):

  • resize: the whole volume is resized to fit the target shape while keeping its aspect ratio (in voxels), then centre-padded. Images use area interpolation when shrinking and trilinear when enlarging; masks use nearest neighbour. The spacing then differs per case: it is stored in each case's metadata, in cases.csv and, as the median over all cases, in dataset.json (median_spacing).
  • roi_crop: a crop of the target shape centred on the bounding box of the mask labels --roi_labels, padded where it leaves the volume. Nothing is resampled, so the spacing stays the same. cases.csv records whether the ROI was found (the crop falls back to the volume centre otherwise) and whether it was larger than the crop. With --lesion_label, it also records how much of that label the crop contains.

Details:

  • Images are padded with their per-channel minimum (air for CT) and masks with 0; masks keep their integer type.
  • Every case needs a spacing in its metadata, as written by stage 1. Cases without one are reported as errors and left out.
  • labels.json and splits.json are copied over, keeping only the cases that have an output. The cases.csv columns listed in --case_columns are carried over from the source; missing ones are skipped with a warning.
  • The run resumes from derive_<mode>_log.jsonl. --metadata_only rewrites the metadata files from that log.
glovita_preprocess generic_3d derive roi_crop --src /data/MyData/MyData_1mm --dst /data/MyData/MyData_1mm_roi192 \
  --roi_labels 1 2 --lesion_label 1 --case_columns patient_id
glovita_preprocess generic_3d derive resize --src /data/MyData/MyData_1mm --dst /data/MyData/MyData_1mm_downsampled192

Example: PANORAMA

PANORAMA (pancreatic cancer, portal-venous CT, 1964 cases after excluding MSD and NIH) is preprocessed by the adapter in panorama.py:

  • intensities are clipped below −1024 HU before the z-score;
  • volumes are oriented to LPS;
  • the label is PDAC versus non-PDAC;
  • splits are grouped by patient: a 20% test set plus 5 folds, seed 42;
  • manual masks are used where they exist, automatic ones otherwise;
  • the ROI crop is centred on the pancreas (parenchyma and duct, labels 4 and 5), deliberately not on the lesion, so the crop position does not leak the label; the lesion (label 1) coverage of the crop is recorded as QC.
# raw/zips/batch_{1..4}.zip from Zenodo, raw/panorama_labels from github.com/DIAGNijmegen/panorama_labels
glovita_preprocess panorama extract --raw_dir RAW
glovita_preprocess panorama preprocess --raw_dir RAW --out_dir OUT --spacings 1mm median \
  --num_workers 6 --mem_budget_gb 24 --min_free_gb 10 --abort_free_gb 3
# fixed-size derivatives of PANORAMA_1mm (PANORAMA's ROI labels, lesion QC and cases.csv columns built in)
glovita_preprocess panorama derive roi_crop --src OUT/PANORAMA_1mm --dst OUT/PANORAMA_1mm_pancreas192 --num_workers 5
glovita_preprocess panorama derive resize   --src OUT/PANORAMA_1mm --dst OUT/PANORAMA_1mm_downsampled192 --num_workers 5

These commands reproduce the existing PANORAMA datasets exactly: bit-identical volumes and byte-identical metadata compared with the original scripts. The one exception is the downsampled dataset's dataset.json, which now also contains median_spacing ([2.3437, 2.3437, 2.3437]). PANORAMA_median's spacing comes out as [1.5, 0.7422, 0.7422].