Pipeline Phases (FSM Walk-Through)

A run of the classification pipeline advances through a finite state machine. Each state is a named phase with a single responsibility, and the legal transitions between phases — defined authoritatively in src/atelier/classify/fsm.py — form the workflow that operators see live in the Workflows page and that this document narrates end-to-end.

This page is the operator-facing companion to two deeper references:

• Classification Pipeline — what the pipeline does mathematically (DST, evidence sources, fusion strategies).
• DST Evidence Independence — why each evidence source qualifies for Dempster’s rule of combination.

Read this one first when you need to walk a reviewer through the run shape: which phase produces which artifact, where the iteration loop lives, what makes a run land in CONVERGED versus ERROR.

At a glance

                                     ┌──── revisit ────┐
                                     │                 │
                                     ▼                 │
IDLE → LOADING_VOCAB → DISCOVERING → SAMPLING → LLM_SWEEP → VALIDATING ──┐
                                                                         │
                                                                         ▼
                                                       CLASSIFYING → FUSING → EVALUATING → CONVERGED
                                                                                                │
                                                                                          (any phase)
                                                                                                ▼
                                                                                              ERROR

The arrow back from VALIDATING to LLM_SWEEP is the iteration loop; it’s the heart of the algorithm and is described in Iteration loop below.

Phases in execution order

The FSM defines two states that the standard inference run does not visit — GENERATING_SYNTH and TRAINING. These belong to the offline synth-corpus generation + SVM-training flow that produces the bundled SVM artifact (legacy filename svm_frontier.pkl retained on disk for backward compatibility with older run directories), and are reachable from SAMPLING only on the explicit synth-generate code path.

Iteration loop: LLM_SWEEP ⇄ VALIDATING is the algorithm

The single most important thing to internalize when reviewing the pipeline: LLM_SWEEP and VALIDATING are not two separate one-shot phases — they form an iteration loop, and the loop is the convergence algorithm.

Each cycle:

1. LLM_SWEEP labels (or re-labels) the directly-targeted column set on iteration 1, the disagreement set on iterations 2…N.
2. VALIDATING runs ML re-validation, computes per-column belief, plausibility, and conflict under the parent-aware DST frame, and identifies the next disagreement set.
3. The loop exits when one of the convergence criteria is satisfied; otherwise it re-enters LLM_SWEEP.

The Workflows page draws this as a purple dashed back-edge from VALIDATING to LLM_SWEEP precisely because the geometry teaches the algorithm: this is bootstrapping with active-learning revisit, not a linear pipeline.

The driver of the loop is configurable:

• Programmatic (default): pipeline._llm_revisit picks revisit candidates from _identify_disagreements and _identify_uncertain_columns.
• Agent-driven (capability flag): the Agent Convergence skill replaces the programmatic driver — Claude chooses revisit candidates and decides when to declare convergence via tool calls.

Convergence criteria

A run reaches CONVERGED when any of the following holds at the end of an iteration in the LLM_SWEEP ⇄ VALIDATING loop:

Conflict K (Dempster’s rule’s normalization mass) is diagnostic, not the gating signal. Earlier iterations of the design framed K as the convergence headline; that framing was retired in commit bd7de2c and matters for any review of older docs or telemetry that still leads with K.

ColumnSample canonical form

The pipeline’s data-model invariant: ColumnSample.name is always the bare column identifier — table-relative, free of any f"{table_name}." prefix. Cross-table identity uses the qualified_name property (f"{table_name}.{name}") for dict keying.

This invariant is enforced in __post_init__ and validated at every source boundary:

• Hive sampler (sampler._strip_table_qualifier) — strips the f"{table_name}." qualifier that Hive’s JDBC driver returns from SELECT * FROM db.table.
• Meta-tagging CSV (meta_tagging_source) — strips the same prefix from CSV headers that encode the table name.
• Synth, OOTB sample, fixtures — produce bare names by construction.

Any new source path that produces qualified names will trip the __post_init__ invariant at construction time with a clear diagnostic, rather than letting them silently propagate into the embedding text — where a repeated table-name prefix in the column name and the sibling list would drown the actual column signal in table-theme noise and produce table-wide misclassification.

Optional capability skills

Three skills attach to specific phases and are gated by the corresponding capability flag. Each renders only when its flag is enabled in /api/status config.

The three skills are visible as orange dashed nodes in the Workflows page when their flags are enabled, attached to their host phases. This is the registry-MVP shape — when we hit roughly six surfaces it graduates to a backend /api/skills endpoint reading from a real registry rather than a hand-coded list.

Phase ↔ artifact map

For an end-to-end review of any single run, here’s what’s recoverable from build/results/{run_id}/, indexed by the phase that produced it:

Where to look in code

State transition reference

Authoritative state-transition table (from fsm.py:_TRANSITIONS):

SAMPLING → CLASSIFYING (skipping LLM_SWEEP) is the path used by Extend runs, where ML-only inference is desired because the LLM has already classified an earlier corpus and the artifacts are being applied to a new dataset. LLM_SWEEP → CLASSIFYING (skipping VALIDATING) is the “first-sweep convergence” path on small corpora that don’t need iteration.