Data cleaning

Cleaning remediates quality findings before versioning, packaging, or training.

Where it fits

Import → Label → Clean → Version / Package → Train

Modality

Typical actions

Image

Remove missing files, exact/near duplicates, blurry images; fix annotated flags; fix split leakage

Text / LLM

Strip whitespace, drop empty / duplicate text, fix invalid preference pairs

Tabular

Drop duplicate rows; fix annotated flags

Audio / Video

Drop samples without a media URI

UI

Project → Quality & Clean tab:

  1. Run checks — quality report + training readiness

  2. Preview — dry-run of selected cleaning actions

  3. Apply cleaning — remediates (may delete rows)

Default actions are pre-selected per modality.

API

GET  /api/cleaning/actions?modality=image
POST /api/projects/{id}/cleaning/preview
POST /api/projects/{id}/cleaning

Body:

{
  "actions": ["remove_exact_duplicates", "fix_annotated_flags"],
  "dry_run": false,
  "quality": { "blur_threshold": 50, "near_dup_hamming": 5 }
}

Omit actions to use modality defaults.

Pipelines

Executable step type: data_cleaning.

- id: clean
  type: data_cleaning
  depends_on: [validate]
  params:
    actions: null   # modality defaults
    dry_run: false

The example PPE pipeline runs image_validation → data_cleaning before splits/train.

Training gate

  • CV train already requires readiness (require_ready).

  • Text / tabular / audio / video / LLM adapters now use the same readiness gate and benefit from cleaning first.

Set require_ready: false or quality.bypass: true to skip.