Monte Carlo Sampling

At small corpus sizes (< 200 columns), every column receives direct LLM classification. As the corpus scales to thousands or millions of columns, this becomes prohibitively expensive. Monte Carlo stratified sampling selects a balanced subset (balanced coverage of the semantic landscape, not corpus-representativeness) for direct LLM inference and propagates labels cheaply via embedding similarity to the remainder.

This is a zero-cost optimization: when in passthrough, the pipeline behaves identically to before. The MC layer is passthrough (classifies every column) whenever total < min_corpus_size or `sample_fraction

= 1.0 (monte_carlo.py). Because the shipped default is sample_fraction = 1.00, MC is passthrough at every corpus size out of the box; sub-sampling engages only once an operator lowers sample_fractionbelow1.0on a corpus larger thanmin_corpus_size`.

Three-Phase MC Layer

The MC layer operates between SAMPLING and LLM_SWEEP in the existing pipeline. No new FSM states — it runs as sub-phases.

SAMPLING
  ├─ [existing] Extract features for all columns
  ├─ Pre-classify: cheap M0 evidence (name, pattern, cosine) — no LLM
  ├─ Stratify: group by preliminary category + uncertainty
  └─ Select MC sample: balance-first within strata (uniform today; importance weighting is roadmap)

LLM_SWEEP
  ├─ [existing] LLM classifies the MC sample (not all columns)
  └─ Propagate: extend labels to remaining corpus via embedding similarity

VALIDATING
  └─ [existing] Full 6-source DST on ALL columns
      (propagated labels enter as discounted LLM evidence)
      → High-gap / low-belief propagated columns escalate to revisit

Phase 1: Pre-Classification

Run M0 evidence sources only (no LLM, no ML models). For each column:

• Name matching → best category + mass
• Pattern detection → matched categories
• Cosine similarity → top-K categories + scores

Returns a preliminary category code + confidence for every column. Uses the existing name_match_to_mass(), pattern_to_mass(), classify_cosine() functions from the pipeline.

Phase 2: Stratification

Partition columns by their preliminary category code:

• Rare strata (< 2 x min_per_stratum members): fully sampled
• UNRESOLVED stratum (M0 sources disagree or low confidence): fully sampled
• Normal strata: proportional allocation over a min_per_stratum floor

Phase 3: Sample Selection

Within each normal stratum, columns are selected by uniform random sampling without replacement, over the proportional allocation and the min_per_stratum floor. The rare/UNRESOLVED 100% guarantee plus the floor are what make the sample balance-first.

Roadmap — importance weighting. The intended steering lever is to bias within-stratum selection toward low-confidence / ambiguous columns by w = (1 - confidence) × (1 + uncertainty) (where confidence = max cosine similarity and uncertainty = 2nd-best/1st-best ratio). This is not yet wired: _importance_sample() is a uniform placeholder because the per-column confidence/ambiguity are not threaded into the selector.

Total budget: min(max_sampled_columns, total × sample_fraction)

The fractions used in the scaling table below (e.g. 15%) illustrate an active-MC configuration with sample_fraction = 0.15; recall the shipped default is 1.00 (passthrough — see the intro).

Label Propagation

After the LLM sweep on the sampled subset:

1. For each propagation column, find the nearest directly-classified column by cosine similarity (global today — every directly-classified column is a candidate source; stratum-local search is a planned scaling optimization, not current behavior)
2. If similarity >= propagation_threshold: assign same label with discounted confidence
3. If similarity < threshold: column gets no LLM evidence in DST

Propagated labels enter DST fusion with a higher discount factor (0.30 vs 0.10 for direct LLM) — they carry less evidential mass. If M0 sources disagree with the propagated label, conflict K rises and the existing targeted-revisit loop automatically escalates the column for direct LLM classification.

Why This Works with DST

The evidence fusion framework makes MC sampling robust:

• Propagated evidence carries less mass (more goes to Theta/ignorance)
• M0 agreement with propagated label → high belief, narrow gap (good)
• M0 disagreement with propagated label → wide gap → revisit-via-LLM
• Escalation is automatic — no special MC-aware revisit logic needed

Scaling Projections

GitTables corpus: 1.7M tables today, 10M+ near-term. Average 8-12 columns per table = 15M-120M columns at full scale.

The table below assumes an active-MC configuration with sample_fraction = 0.15 (the shipped default is 1.00 — see the note above; “Passthrough” behaviour is also what the default produces at any corpus size).

At the max_sampled_columns=500 cap, stratified sampling still guarantees every category stratum gets at least min_per_stratum=3 exemplars (rare / UNRESOLVED strata at 100%). Naive uniform sampling at 500/15M — without the stratum floor — would miss rare categories entirely.

Scale-Critical Design Decisions

• Embedding computation: batch GPU encoding at ~2,768 texts/s (RTX 4090); 15M columns takes ~90 minutes. One-time cost, GPU-parallelizable.
• Propagation scope: propagation is global today (every sampled column is a candidate source). Restricting similarity search to within-stratum to limit memory/compute at scale is a planned optimization (not yet implemented).
• Memory: 15M columns × 200B = ~3GB for metadata; 15M × 1.5KB = ~22GB for embeddings. Requires streaming/chunked processing.
• Escalation budget: ~50-100 additional direct-LLM calls from revisit. Total LLM call budget: ~600 calls for a 15M-column corpus.

Configuration

classify {
  monte_carlo {
    min_corpus_size = 200              # Below this, classify everything
    min_corpus_size = ${?ATELIER_MC_MIN_CORPUS_SIZE}
    sample_fraction = 1.00             # Fraction directly classified by LLM (shipped default: classify all)
    sample_fraction = ${?ATELIER_MC_SAMPLE_FRACTION}
    min_per_stratum = 3                # Minimum samples per category stratum
    max_sampled_columns = 500          # Hard cap on directly-classified columns
    max_sampled_columns = ${?ATELIER_MC_MAX_SAMPLED}
    propagation_threshold = 0.80       # Cosine sim for propagation
    propagation_threshold = ${?ATELIER_MC_PROPAGATION_THRESHOLD}
    propagation_discount = 0.30        # LLM mass discount for propagated labels
  }
}

Module Structure

src/atelier/classify/monte_carlo.py
├── MCConfig          — Frozen dataclass with from_cfg() factory
├── PreClassification — Per-column M0 result (code + confidence + uncertainty)
├── Stratum           — Column group by preliminary category
├── MCPlan            — Sampling plan (sampled + propagation sets)
├── pre_classify()    — Run M0 evidence for all columns
├── stratify()        — Group by preliminary category + uncertainty
├── select_sample()   — Balance-first selection within strata (uniform today; importance weighting roadmap)
└── propagate_labels() — Embedding-similarity label extension