Detail Pipeline of Student Training -- sanoTTS

1. Execution Command & Configuration

The entire pipeline is executed via a single master script.

Standard Execution Command:

python3 tools/train_voice_from_piper.py --voice pl_PL-gosia-medium \
  --text data/textsets/polish-distill-a-v1/pl_PL.train.jsonl \
  --size standard --profile shipped

Practical Tip: Before committing hours to a full training run, perform a cheap plumbing pass using --profile smoke. If the smoke profile completes S0–S11 without errors, relaunch with --profile shipped for the actual production run. Note that --profile shipped and --size standard are the default values, so the command works identically without explicitly declaring them.

Model Size Configuration (--size)

Defines the architectural width of the student models. Parameters scale with the square of these widths.

Training Profile Configuration (--profile)

Defines the number of training steps per stage.

2. Pipeline Flow & Summary

One-Line Flow: Preflight → S1 (Teacher Data) → S2 (Decoder Cut) → S3 (Duration Student) & S4 (Acoustic Student) → S5 (Decoder Init) & S6 (Signatures) → S7 (Decoder Train) → S8 (Z-Mix) → S9 (Joint Fine-tune) → S10 (Quality Gate) → S11 (Package).

High-Level Summary: S1 creates teacher supervision → S3/S4 learn the acoustic side → S5/S6/S7/S8 build the decoder → S9 synchronizes everything → S10 evaluates it → S11 packages the final model.

3. Stage-by-Stage Breakdown

Preflight

Script: make_multilang_text_distill_corpus.py (if --auto-text is used, otherwise skipped). Primary Function: Validates the teacher/model setup before any compute begins. It checks:

1. The teacher’s phoneme type (espeak, pinyin, etc.).
2. Whether eSpeak can phonemize the language.
3. The decoder width is exactly 256 channels.
4. It is a single-speaker model.
5. Sample rate is valid. Note: The provided pl_PL.train.jsonl is not examined during preflight. Key Output: Preflight passed (console output).

S1: Teacher Data Extraction

Script: build_piper_vits_roota_probe_pack.py Primary Function: Queries the teacher model to extract deterministic ground-truth data (phoneme IDs, durations, latents, audio) for training and evaluation. Key Arguments:

• --model: Teacher Piper ONNX model.
• --config: Teacher model configuration JSON.
• --source-jsonl: Input text corpus.
• --max-rows / --skip-rows: Controls row selection for train/eval splits.
• --tensor-mode: full (all debug tensors), decoder (decoder-specific), or acoustic (latent-specific). Key Output: 4 Data Packs (smoke, eval, acoustic, decoder).

S2: Decoder Cut

Script: extract_piper_vits_decoder_cut.py Primary Function: Slices the decoder portion out of the teacher’s full ONNX model to act as a perfect “oracle” reference during student training. Key Arguments:

• --model: Full teacher Piper ONNX model.
• --pack-dir: S1 smoke pack directory for validation.
• --latent-name: ONNX tensor name where the model is cut (defaults to generator_input).
• --latent-channels: Number of channels in the latent tensor (defaults to 192). Key Output: decoder-cut/*-decoder-from-generator-input.onnx

S3: Duration Student Training

Script: train_roota_piper_duration_student.py Primary Function: Trains a small neural network to predict phoneme durations (timing), mimicking the teacher’s duration model. Inputs: phoneme_ids and teacher target w_ceil duration tensors from the train-acoustic pack. Outputs: log_duration for each phoneme (padding masked out). Key Output: duration/duration-student.pt

S4: Acoustic Student Training

Script: train_roota_piper_latent_student.py Primary Function: Trains the acoustic student to predict hidden acoustic “latents”. Uses adversarial loss to prevent the audio from sounding over-smoothed or robotic. Inputs: phoneme_ids, w_ceil (durations), and generator_input (teacher latent targets) from the train-acoustic pack. Oracle Reference: Uses the sliced teacher decoder (--decoder) for feature matching to ensure student latents produce good audio. Key Output: acoustic/latent-student.pt

S5: Decoder Initialization & Parity

Script: verify_piper_decoder_torch_parity.py Primary Function: Verifies the sliced ONNX decoder matches the teacher’s output, then extracts and exports channel-sliced initial weights for the student decoder. Processing: Slices required decoder channels and initializes student weights based on importance. Key Output: parity/piper-decoder-teacher.pt (channel-sliced, importance-initialized weights).

S6: Signature Pack Building

Script: build_piper_vits_decoder_signature_pack.py Primary Function: Dynamically reads the cut decoder’s ONNX graph to find specific activation nodes, building a “signature pack” to guide the student’s internal learning. Five Required Signature Keys:

1. latent: Input to the decoder (acoustic latent from S4).
2. stage0_mix: Output of the first mixing/convolution stage.
3. stage1_mix: Output after the first upsampling stage.
4. stage2_mix: Output after the second upsampling stage.
5. pre_tanh: Activation immediately before the final tanh layer.
6. audio: Final generated waveform. Key Output: signatures/summary.json and captured activation .npy files.

S7: Decoder Student Training (Pass 1)

Script: train_roota_piper_decoder_student.py Primary Function: Trains the compact student decoder, initialized from the teacher’s weights, guided by the signature pack to mimic internal representations. Dependencies: S1d, S2, S4, S5, S6. Key Inputs:

1. train512-decoder pack (S1d).
2. eval128 pack (S1b).
3. decoder-cut.onnx (S2).
4. piper-decoder-teacher.pt (S5).
5. latent-student.pt (S4).
6. signatures/ directory (S6). Key Output: decoder/decoder-student.pt

S8: Decoder Student Training (Pass 2 / Z-Mix)

Script: train_roota_piper_decoder_student.py (Run 2) Primary Function: “Z-Mix” Adaptation. Retrains the decoder, feeding it the acoustic student’s imperfect latents 50% of the time. This “closes the interface” and prevents brittle failures. Comparison with S7:

• --init-decoder-checkpoint: Uses S7’s output (S7 did not use this).
• --teacher-init-checkpoint: Not used (S7 used S5’s output).
• --acoustic-latent-mix-prob: Set to 0.5 (S7 did not use this).
• --adv-start-step: 1000 (S7 was 1500).
• --steps: 2500 (S7 was 2800).
• --lr: 1e-4 (S7 was 2e-4). Key Output: decoder-zmix/decoder-student.pt

S9: Joint Fine-Tuning

Script: train_roota_joint_z_finetune.py Primary Function: Finetunes the acoustic and decoder models together to ensure they are perfectly synchronized and optimized as a single, cohesive pipeline. Inputs: Acoustic checkpoint (S4), Decoder checkpoint (S8), Teacher decoder (S2), Training data (S1d). Key Output: Final joint/latent-student.pt & joint/decoder-student.pt.

S10: Quality Gate

Script: run_roota_feedback_loop.py Primary Function: Evaluates the final stack against the held-out eval pack, generates sample audio, and computes objective metrics (e.g., SCOREQ) to verify quality. Inputs: Eval pack (S1b), Acoustic/Decoder/Duration checkpoints (S9, S3), Teacher model, Parameter budget (--param-budget 2000000), Optional eval textset. Key Output: feedback/feedback-loop-summary.json, per-lane score summaries, and generated WAV audio samples.

S11: Export Package

Script: export_roota_self_contained_package.py Primary Function: Bundles the final trained checkpoints, Piper config, sample rate, and metadata into a single, ready-to-deploy directory. Key Output: package/ directory containing manifest.json and all inference-only weights.

4. S1 Data Packs Details

The teacher model generates four distinct data packs used throughout the pipeline.

Pack Directory Contents: Each pack directory contains:

• rows.json: Metadata (processed row IDs, durations).
• phoneme_ids/: Piper phoneme ID sequences.
• w_ceil/: Duration tensors (how long each phoneme lasts).
• generator_input/: Latent/acoustic tensors produced before the decoder.
• audio/: WAV audio generated by the teacher.
• path/, attn_mask/, etc.: Optional alignment/debug tensors depending on --tensor-mode.

5. Verification Checklist

Use this checklist to verify the pipeline executed correctly at each stage.

6. Final Artifact Directory Structure

Upon completion, all outputs are organized within the voice-specific artifact directory:

artifacts/
└── voices/
    └── pl_PL-gosia-medium/
        ├── acoustic/          (S4 outputs)
        ├── dashboard/         (Serving/UI assets)
        ├── decoder/           (S7 outputs)
        ├── decoder-cut/       (S2 outputs)
        ├── decoder-zmix/      (S8 outputs)
        ├── duration/          (S3 outputs)
        ├── eval128/           (S1b pack)
        ├── feedback/          (S10 outputs)
        ├── joint/             (S9 outputs)
        ├── logs/              (Execution logs)
        ├── package/           (S11 final shippable output)
        ├── parity/            (S5 outputs)
        ├── signatures/        (S6 outputs)
        ├── smoke12/           (S1a pack)
        ├── train-acoustic/    (S1c pack)
        └── train512-decoder/  (S1d pack)