# Best-results study: V1 and Netherlands 2017

**Run date:** 2026-09-15  
**Status:** complete retrospective development study; not prospective product validation

## Executive result

The strongest new V1 result comes from the field photos, not the six-band
sensor. A fixed multiscale RGB PLS-DA candidate reaches **ROC-AUC 0.907** and
**average precision 0.936**. A fully nested selector that accounts for choosing
the RGB representation/model reaches **ROC-AUC 0.891**. The reproduced spectral
baseline remains **0.784**. Spectral plus RGB features do not improve on RGB
alone.

The strongest defensible Netherlands candidate uses the distribution of canopy
pixels plus a label-blind comparison with nearby plants. It reaches
**ROC-AUC 0.902** and **average precision 0.244**, versus **0.890 / 0.195** for
the reconstructed mean-SNV baseline. A fixed two-model ensemble reaches
**ROC-AUC 0.911**, but that ensemble was identified after examining the model
table; the fully nested representation selector is the safer selection-adjusted
number at **0.898**.

These gains do not change the product conclusion. V1 still has visual—not
laboratory—labels, and the Netherlands data still has only 68 independent PVY
plants, an invalid dark reference, a clipped white reference, and one unusually
hard row. Precision at high recall remains the limiting metric.

## Evaluation contract

- V1 has one row per biological plant. Every hyperparameter is selected inside
  each outer fold. Fixed candidates use 10 repeats of five-fold CV; the
  representation/model selector uses five repeats of five-fold fully nested CV.
- Netherlands keeps both dates from a biological plant in the same fold. Fixed
  candidates use three independently shuffled five-fold grouped evaluations,
  with grouped inner CV for hyperparameters.
- Confidence intervals use bootstrap resampling at the biological-plant level.
- No site, row, date, variety, label prevalence, or plant identifier is supplied
  as a predictor.
- High-recall operating points are retrospective descriptions of out-of-fold
  scores, not frozen prospective thresholds.

## V1

### Results

| Candidate | ROC-AUC | 95% CI | Average precision | 95% CI |
|---|---:|---:|---:|---:|
| **RGB multiscale PLS-DA** | **0.907** | 0.839–0.959 | **0.936** | 0.881–0.975 |
| RGB central-crop LDA | 0.904 | 0.844–0.958 | 0.939 | 0.895–0.974 |
| RGB + spectral LDA | 0.898 | 0.830–0.957 | 0.936 | 0.887–0.973 |
| **Fully nested RGB selector** | **0.891** | 0.822–0.947 | **0.931** | 0.881–0.966 |
| Reproduced spectral LDA baseline | 0.784 | 0.689–0.880 | 0.842 | 0.734–0.926 |

The fixed RGB PLS model's best observed point at least 90% recall is **84.1%
precision at 93.0% recall**: 53 TP, 10 FP, 4 FN, 32 TN. The fully nested RGB
selector gives **81.2% precision at 91.2% recall**: 52 TP, 12 FP, 5 FN, 30 TN.
Those values apply only to this deliberately balanced 57/42 sample, not to a
commercial field prevalence.

Whole-site transfer of the fixed RGB candidate remains asymmetric:

| Train | Test | ROC-AUC | Average precision |
|---|---|---:|---:|
| UI | ICIA | 0.774 | 0.834 |
| ICIA | UI | 0.933 | 0.961 |

### What the V1 gain means

The photos contain the visual symptoms used to assign the target label, so this
is strong evidence that compact RGB summaries can reproduce the field symptom
call. It is not evidence of presymptomatic or latent PVY detection. The result
supports the product vision's RGB-plus-spectral acquisition strategy, but it
also makes the next experiment clearer: RGB must be evaluated against ELISA or
PCR labels, including visually healthy infected plants and visually abnormal
PVY-negative plants.

Adding spectra to RGB does not improve the result. That is consistent with the
known V1 sensor failures and argues against spending more modeling effort on
these six-band readings. The best next improvement is better ground truth and
measurement geometry, not a larger classifier.

## Netherlands 2017

### Reconstruction audit

The original processed CSV and ingest scripts are absent from the current
worktree. The new extractor reads the immutable HDF5 bundle and support ZIP.
Projection onto the crop-row axis reproduces the archived **251,340 assigned
frames exactly**, along with **132 PVY visits from 68 PVY plants**.

The reconstructed model cohort has 3,371 visits from 1,704 plants—25 healthy
visits and seven plants more than the legacy report's 3,346 / 1,697. The missing
legacy script prevents proving the last exclusion difference. Results in this
folder therefore do not silently replace the settled canonical numbers.

### Results

| Candidate | ROC-AUC | 95% CI | Average precision | Precision at ≥90% recall |
|---|---:|---:|---:|---:|
| Mean SNV LDA reconstruction | 0.890 | 0.856–0.921 | 0.195 | 18.9% |
| Median SNV PLS-DA | 0.895 | 0.865–0.921 | 0.192 | 14.5% |
| Pixel-distribution LDA | 0.897 | 0.862–0.927 | 0.217 | 15.5% |
| **Distribution + local-context LDA** | **0.902** | **0.867–0.932** | **0.244** | **17.3%** |
| Fully nested representation selector | 0.898 | 0.861–0.928 | 0.238 | 15.1% |
| Fixed Montana-style pixel attention | 0.887 | — | 0.260 | 11.7% |
| Exploratory fixed ensemble | **0.911** | 0.882–0.936 | 0.225 | **18.7%** |

The fixed ensemble averages distribution/local LDA and median-spectrum PLS-DA.
It is useful evidence that their errors differ, but 0.911 is exploratory because
the pairing was chosen after viewing the candidate results. The nested selector
is the correct number when accounting for representation choice.

At the distribution/local candidate's ≥90% recall point, the selected cohort
produces 119 TP, 568 FP, 13 FN, and 2,671 TN. That is **17.3% precision at 90.2%
recall** at 3.9% prevalence. This improves ranking, but it is still far from an
automatic kill decision.

Optimizing inner folds for average precision instead of ROC-AUC yields AP
**0.368**, but precision at 90% recall falls to 10.1%. It concentrates positives
near the top of the list while degrading the high-recall tail. For this product,
AP alone is not an adequate model-selection target.

### Transfer and robustness

| Held-out row | ROC-AUC | AP |
|---|---:|---:|
| row1 | 0.890 | 0.603 |
| row2 | 0.994 | 0.864 |
| row3 | 0.911 | 0.680 |
| **row5** | **0.666** | **0.135** |
| row6 | 0.948 | 0.810 |
| row7 | 0.908 | 0.204 |

Row5 remains the binding robustness failure and becomes worse with the richer
model. A higher pooled AUC must not hide that.

Temporal transfer preserves the original asymmetry: training on 2017-07-03 and
testing 2017-06-27 gives **0.885**, while training on 2017-06-27 and testing
2017-07-03 gives **0.705**. These are transport checks, not independent tests,
because the same plants can occur on both dates.

The Montana-style attention network does not improve Netherlands ranking. It
raises AP relative to the mean-spectrum baseline, but its 0.887 AUC and 11.7%
precision at 90% recall are below the simpler distribution/local LDA. Retaining
within-plant variation helps; a learned pixel-attention layer is not necessary
for this cohort.

### What one held-out inference looks like

The Netherlands source does **not** provide pixel masks, and our model is not a
pixel-wise classifier.  Each ground-truth row provides `PlantName`, `RijName`,
a plant GPS point and four GPS box corners.  Those coordinates select a short
along-row interval of consecutive pushbroom frames.  We then apply a fixed,
label-blind NDVI > 0.60 canopy filter to the 512 cross-track samples and reduce
the retained spectra to plant-level quantiles, spread and nearby-plant context.
The best LDA emits one score for the whole plant visit.

[`interpretability/netherlands-heldout-example.png`](interpretability/netherlands-heldout-example.png)
shows that path for `20170703:row6:PVY-153`.  Both dates from this biological
plant were excluded from training.  The GPS interval contributes 71 scan lines
and 19,625 retained vegetation pixels; the fitted outer-fold LDA scores it
0.99999 (uncalibrated), at the 99.9th percentile of its 675-visit test fold.
This is an intentionally legible true-positive example, not an unbiased sample
of cases or an additional accuracy estimate.

[`interpretability/v1-heldout-example.png`](interpretability/v1-heldout-example.png)
does the same for the V1 RGB path: the original field photo, the three crop
scales, the label-blind vegetation mask and the repeated out-of-fold score
distribution.  Its target remains the supplied visual symptom call, not a
laboratory PVY result.

## Decision implications

1. **Add RGB to the next target-platform collection.** It is the strongest V1
   modality, cheap, and already part of the product vision.
2. **Do not interpret the V1 RGB number as PVY diagnosis.** Its target is a
   visible symptom call. The next study needs plant-level laboratory truth and
   hard negative stresses.
3. **Retain within-plant distributions.** Netherlands improves when quantiles,
   spread, and local comparison are preserved instead of collapsing each plant
   to one mean spectrum.
4. **Keep models small.** LDA/PLS outperform the transferred attention network.
   With 68 independent positive plants, data quality and cohort diversity still
   dominate architecture.
5. **Use recall-constrained precision as the primary selection rule.** ROC-AUC
   and AP can both improve while the operational high-recall point gets worse.
6. **Treat row5 and cross-date transfer as gates for V2.** A pooled score above
   0.9 is not enough if one row or growth stage collapses.

## Reproduction

```bash
venv/bin/python -m pip install -r best_results/requirements.txt
venv/bin/python best_results/extract_v1_rgb.py
venv/bin/python best_results/extract_netherlands.py
venv/bin/python best_results/confirm_v1.py
venv/bin/python best_results/confirm_netherlands.py
venv/bin/python best_results/run_netherlands_attention.py --epochs 20 --bag-pixels 64
MPLCONFIGDIR=/tmp/pvy-mpl-cache venv/bin/python best_results/make_interpretability.py
```

Machine-readable metrics and out-of-fold predictions are under
`best_results/outputs/`. The 147 MB Netherlands pixel cache is reproducible and
ignored by the folder-local `.gitignore`; raw measurements remain untouched.