Skip to content
Multivariate Lab

Assignment 3 · evaluation added in 2026

How good are the rainfall classifiers, really?

The 2023 report compared three classifiers by their errors on 41 test stations. With so few stations, one or two errors either way is noise. Here every pipeline is re-estimated by repeated cross-validation on the 150 training stations, on identical folds, so the comparisons are paired and every number comes with an interval. The original results are unchanged and still shown on the rainfall page.

CV folds
10×5
CV folds
train stations
150
train stations
test stations
41
test stations

Design

Same folds, whole pipeline refitted

Each training fold is treated as if it were the only data: standardisation, PCA or PLS and every tuning step happen inside the fold, then the held-out stations are predicted once. For the corrected pipelines nothing about the held-out stations leaks into the model. The 2023-settings logistic pipeline keeps q = 27, which was chosen in 2023 on all 150 stations; that can only favour it, so the gain from fixing it is, if anything, understated.

Why corrected intervals?
The 50 fold results are not independent: any two training folds share about three quarters of their stations. A plain paired t-test treats them as independent and is far too confident. The Nadeau–Bengio correction scales the variance of the mean by (1/J + ntest/ntrain) = 0.27 instead of 1/J = 0.02, so the intervals are about 3.7 times wider.

Accuracy

Cross-validated accuracy with 95% intervals

Mean accuracy over the 10 repeats (each repeat predicts every station once), with a Wilson interval at n = 150 stations. Green: the corrected methods; ochre: the same pipelines with the 2023 settings; grey: always predicting South.

Accuracy by pipeline
dashed line: majority-class baseline (61.3%)
50%60%70%80%90%100%cross-validated accuracymajority classPCA + logistic: 94.7% (89.9–97.3)PCA + logistic94.7% (89.9–97.3)Random forest: 92.4% (87.0–95.7)Random forest92.4% (87.0–95.7)PLS tree: 91.1% (85.5–94.7)PLS tree91.1% (85.5–94.7)PCA + logistic (2023): 83.7% (77.0–88.8)PCA + logistic (2023)83.7% (77.0–88.8)Random forest (2023): 92.5% (87.2–95.8)Random forest (2023)92.5% (87.2–95.8)PLS tree (2023): 67.6% (59.8–74.6)PLS tree (2023)67.6% (59.8–74.6)Majority: 61.3% (53.3–68.8)Majority61.3% (53.3–68.8)

The Wilson interval at n = 150 reflects which stations happen to be in the data; the range over repeats reflects only how they are split into folds. Neither covers everything (for example, a different set of stations from the same regions), so read the intervals as a lower bound on the real uncertainty.

Paired comparisons

Which differences are real?

Accuracy differences in percentage points, fold by fold, with the corrected 95% interval. Slate: corrected method against corrected method; grey: against the majority-class baseline; ochre: the 2023 bug fixes (corrected minus as submitted). The page runs nine tests, so p-values are Holm-adjusted within each family of three: method vs method, method vs baseline and corrected vs 2023. The intervals are ordinary 95% intervals, not simultaneous ones.

Paired accuracy differences
dashed line: no difference
−200+20+40difference in accuracy (percentage points)PCA + logistic − Random forest: +2.4 (−1.5 to +6.2)PCA + logistic − Random forest+2.4 (−1.5 to +6.2)PCA + logistic − PLS tree: +3.6 (−1.8 to +9.1)PCA + logistic − PLS tree+3.6 (−1.8 to +9.1)Random forest − PLS tree: +1.3 (−3.5 to +6.1)Random forest − PLS tree+1.3 (−3.5 to +6.1)PCA + logistic − majority: +33.4 (+29.6 to +37.3)PCA + logistic − majority+33.4 (+29.6 to +37.3)Random forest − majority: +31.1 (+26.8 to +35.3)Random forest − majority+31.1 (+26.8 to +35.3)PLS tree − majority: +29.8 (+24.4 to +35.1)PLS tree − majority+29.8 (+24.4 to +35.1)PCA + logistic: fixed − 2023: +11.1 (+0.8 to +21.3)PCA + logistic: fixed − 2023+11.1 (+0.8 to +21.3)Random forest: fixed − 2023: −0.1 (−2.4 to +2.2)Random forest: fixed − 2023−0.1 (−2.4 to +2.2)PLS tree: fixed − 2023: +23.6 (+9.6 to +37.6)PLS tree: fixed − 2023+23.6 (+9.6 to +37.6)
The three corrected methods are statistically indistinguishable
PCA + logistic regression had the highest accuracy in every one of the 10 repeats, yet its advantage over the forest (+2.4 pp) has a corrected interval that includes zero. A consistent ranking across repeats is not evidence on its own: the repeats reuse the same 150 stations. On the test set the corrected methods disagree on at most 2 stations, which no test can resolve.
One of the three 2023 bugs clearly mattered
Fixing the PLS test scaling adds +23.6 pp (+9.6 pp to +37.6 pp, Holm p = 0.004). Fixing the 2023 settings of the logistic model points the same way, +11.1 pp (+0.8 pp to +21.3 pp), but does not survive the adjustment for testing three fixes (Holm p = 0.069); its q = 27 was tuned on all 150 stations, which can only flatter the 2023 version. The m bug in the random forest changed nothing (−0.1 pp), as DR-001 explains.

Errors by class

Where the mistakes are

Predictions summed over the 10 repeats, with the recall of each class (Wilson interval at the class size: 58 North, 92 South). The logistic model and the forest recognise southern stations more reliably than northern ones (the smaller class); the PLS tree is more balanced. The 2023 PLS scaling bug sends 48% of the southern stations to the wrong class.

PCA + logistic

PCA + logistic: predictions summed over 10 repeats of cross-validation (1500 in total)
actual
NorthSouth
predicted North53130
predicted South49890
North recall
91.6% (81.6–96.4%)
South recall
96.7% (90.8–98.9%)

1,500 predictions: each of the 150 stations, once per repeat.

Random forest

Random forest: predictions summed over 10 repeats of cross-validation (1500 in total)
actual
NorthSouth
predicted North50943
predicted South71877
North recall
87.8% (76.9–93.9%)
South recall
95.3% (88.9–98.1%)

1,500 predictions: each of the 150 stations, once per repeat.

PLS tree

PLS tree: predictions summed over 10 repeats of cross-validation (1500 in total)
actual
NorthSouth
predicted North53487
predicted South46833
North recall
92.1% (82.2–96.7%)
South recall
90.5% (82.8–95.0%)

1,500 predictions: each of the 150 stations, once per repeat.

PLS tree (2023)

PLS tree (2023): predictions summed over 10 repeats of cross-validation (1500 in total)
actual
NorthSouth
predicted North537443
predicted South43477
North recall
92.6% (82.9–97.0%)
South recall
51.8% (41.8–61.8%)

1,500 predictions: each of the 150 stations, once per repeat.

Out-of-bag vs cross-validation

Can the forest's own OOB error be trusted?

A random forest estimates its own error from the stations each tree did not see. Inside every outer fold, the refitted forest's OOB error (computed on the training fold only) is compared with its error on the held-out stations.

OOB estimate against held-out error
mean OOB 6.97%, mean held-out 7.60%
Repeat 1, fold 1: OOB 8.4%, held-out 6.5%Repeat 1, fold 2: OOB 5.0%, held-out 16.1%Repeat 1, fold 3: OOB 5.8%, held-out 3.3%Repeat 1, fold 4: OOB 9.9%, held-out 0.0%Repeat 1, fold 5: OOB 7.4%, held-out 10.3%Repeat 2, fold 1: OOB 5.9%, held-out 9.7%Repeat 2, fold 2: OOB 6.7%, held-out 12.9%Repeat 2, fold 3: OOB 7.5%, held-out 0.0%Repeat 2, fold 4: OOB 4.1%, held-out 10.3%Repeat 2, fold 5: OOB 6.6%, held-out 13.8%Repeat 3, fold 1: OOB 7.6%, held-out 6.5%Repeat 3, fold 2: OOB 7.6%, held-out 6.5%Repeat 3, fold 3: OOB 5.0%, held-out 16.7%Repeat 3, fold 4: OOB 8.3%, held-out 10.3%Repeat 3, fold 5: OOB 8.3%, held-out 3.4%Repeat 4, fold 1: OOB 6.7%, held-out 6.5%Repeat 4, fold 2: OOB 7.6%, held-out 3.2%Repeat 4, fold 3: OOB 6.7%, held-out 16.7%Repeat 4, fold 4: OOB 6.6%, held-out 13.8%Repeat 4, fold 5: OOB 7.4%, held-out 3.4%Repeat 5, fold 1: OOB 6.7%, held-out 9.7%Repeat 5, fold 2: OOB 6.7%, held-out 6.5%Repeat 5, fold 3: OOB 8.3%, held-out 6.7%Repeat 5, fold 4: OOB 5.8%, held-out 6.9%Repeat 5, fold 5: OOB 8.3%, held-out 10.3%Repeat 6, fold 1: OOB 8.4%, held-out 6.5%Repeat 6, fold 2: OOB 7.6%, held-out 6.5%Repeat 6, fold 3: OOB 5.0%, held-out 10.0%Repeat 6, fold 4: OOB 8.3%, held-out 6.9%Repeat 6, fold 5: OOB 5.8%, held-out 10.3%Repeat 7, fold 1: OOB 6.7%, held-out 6.5%Repeat 7, fold 2: OOB 6.7%, held-out 3.2%Repeat 7, fold 3: OOB 9.2%, held-out 3.3%Repeat 7, fold 4: OOB 6.6%, held-out 10.3%Repeat 7, fold 5: OOB 5.8%, held-out 10.3%Repeat 8, fold 1: OOB 7.6%, held-out 6.5%Repeat 8, fold 2: OOB 7.6%, held-out 3.2%Repeat 8, fold 3: OOB 5.8%, held-out 10.0%Repeat 8, fold 4: OOB 6.6%, held-out 6.9%Repeat 8, fold 5: OOB 8.3%, held-out 3.4%Repeat 9, fold 1: OOB 6.7%, held-out 6.5%Repeat 9, fold 2: OOB 5.9%, held-out 9.7%Repeat 9, fold 3: OOB 5.0%, held-out 10.0%Repeat 9, fold 4: OOB 9.9%, held-out 0.0%Repeat 9, fold 5: OOB 5.8%, held-out 6.9%Repeat 10, fold 1: OOB 6.7%, held-out 6.5%Repeat 10, fold 2: OOB 5.9%, held-out 9.7%Repeat 10, fold 3: OOB 7.5%, held-out 3.3%Repeat 10, fold 4: OOB 8.3%, held-out 3.4%Repeat 10, fold 5: OOB 5.8%, held-out 10.3%Repeat 1: mean OOB 7.3%, CV error 7.3%Repeat 2: mean OOB 6.2%, CV error 9.3%Repeat 3: mean OOB 7.3%, CV error 8.7%Repeat 4: mean OOB 7.0%, CV error 8.7%Repeat 5: mean OOB 7.2%, CV error 8.0%Repeat 6: mean OOB 7.0%, CV error 8.0%Repeat 7: mean OOB 7.0%, CV error 6.7%Repeat 8: mean OOB 7.2%, CV error 6.0%Repeat 9: mean OOB 6.7%, CV error 6.7%Repeat 10: mean OOB 6.8%, CV error 6.7%
  • one outer fold (≈30 held-out stations)
  • one repeat (all 150 stations)
  • OOB = CV
Three estimates of the forest's error rate
95% Wilson intervals
0%5%10%15%20%25%misclassification rateOOB, all 150 stations: 6.7% (3.7–11.8)OOB, all 150 stationsm = 57, 5,000 trees6.7% (3.7–11.8)Repeated CV: 7.6% (4.3–13.0)Repeated CV10 × 5 folds, m tuned per fold7.6% (4.3–13.0)Test set: 7.3% (2.5–19.4)Test set3 of 41 stations7.3% (2.5–19.4)
They agree
Per fold, OOB minus held-out error averages −0.6 pp (corrected 95% interval −5.8 pp to +4.5 pp). The OOB error is, if anything, slightly optimistic, which is expected when m is picked by the same OOB error, but it is well within the noise. The test set agrees too; its interval is simply much wider.

Tuning stability

How firmly are q and m determined?

The value each outer fold's own tuning picked. If tuning had found a real optimum, the choices would cluster.

Number of principal components q
leave-one-out CV inside each training fold; ochre: q = 3 chosen on all 150 stations
q = 2: chosen in 7 of the outer folds7q = 3: chosen in 12 of the outer folds12q = 4: chosen in 8 of the outer folds8q = 5: chosen in 10 of the outer folds10q = 6: chosen in 10 of the outer folds10q = 8: chosen in 2 of the outer folds2q = 9: chosen in 1 of the outer folds1
  • chosen in that many of the 50 outer folds
  • q chosen on all 150 stations (3)
Variables tried per split m
lowest OOB error on the grid inside each fold; ochre: m = 57 chosen on all 150 stations
m = 5: chosen in 17 of the outer folds17m = 10: chosen in 7 of the outer folds7m = 19: chosen in 7 of the outer folds7m = 38: chosen in 8 of the outer folds8m = 57: chosen in 3 of the outer folds3m = 76: chosen in 2 of the outer folds2m = 115: chosen in 3 of the outer folds3m = 153: chosen in 3 of the outer folds3
  • chosen in that many of the 50 outer folds
  • m chosen on all 150 stations (57)
2–9
range of q chosen by the folds
small q always wins; the exact value does not matter much
17/50
folds picking the smallest m (5)
8 of the 10 grid values were chosen by at least one fold
0.8%–3.3%
OOB spread across the m grid within a fold
1 to 4 stations: the OOB curve is nearly flat

The PLS tree split on component C1 in all 50 folds; its second split varied. Together with the flat OOB curve this says the rainfall signal is strong and low-dimensional (the summer monsoon), so the exact tuning values are weakly identified and barely affect accuracy. The highest point estimate is PCA + logistic regression at 94.7% (89.9–97.3%), but the three corrected pipelines cannot be told apart, so treat that figure as optimistic for whichever one is picked.

Reproduce

Where these numbers come from

The folds, fits and test-set predictions are produced in R by Rscript scripts/a3_evaluation.R (R version 4.6.1 (2026-06-24); randomForest 4.7.1.2, pls 2.9.0, rpart 4.1.27). It replays the 2023 and corrected fits with their original seeds and stops unless they reproduce the committed confusion matrices exactly. Only per-fold confusion counts, tuning choices, OOB estimates and test-set predictions are published (/data/rainfall-evaluation.json).

Every interval, test and summary on this page is computed from those counts by the TypeScript helpers in src/lib/stats, which are unit-tested against R (prop.test, binom.test, qt, quantile). Method notes, assumptions and limitations are on the methods page.