Methods · decisions · AI use

How the numbers were made

Where the data come from, which methods were used and how they were checked, what the evaluation assumes and where it stops, the model card, the decisions behind the site written up as decision records, and exactly what the optional AI feature does.

Data provenance

Wheat kernels (Assignment 1)

UCI Machine Learning Repository “seeds” (Charytanowicz et al., 2010), CC BY 4.0: 210 kernels, 70 each of Kama, Rosa and Canadian wheat, seven geometric measurements from soft X-ray images. Copied from coursework/assignment-1/Wheat.txt into the app by scripts/a1_reference.R and analysed in your browser.

Rainfall stations (Assignment 3)

Daily average rainfall (365 values) for 150 training and 41 test stations with a North/South label, supplied with the subject. Redistribution terms are unknown, so the site serves only derived quantities: class summaries, projections, importances, confusion counts and predictions (DR-003). The raw files stay in the repository for reproducibility and are read only by the R scripts and the test suite.

Methods

The 2023 R analyses were re-run unchanged (“as submitted”) and with the documented bugs fixed (“corrected”). The upgrade adds uncertainty and an honest comparison around those results; it does not change them.

PCA (wheat)

Eigen-decomposition of the sample covariance or correlation matrix by a TypeScript Jacobi solver, checked against R's eigen and prcomp to 1e-10.

Bootstrap intervals (wheat)

Kernels resampled with replacement (1,000 times by default, seed 90138), the whole PCA redone each time; 95% percentile intervals for eigenvalues, variance shares, ψ and sign-aligned loadings, plus a stability score per component.

Parallel analysis (wheat)

Horn's parallel analysis with a column-permutation null (each variable keeps its distribution, correlations are destroyed), which works for covariance and correlation PCA alike; the normal-data version is verified against an R simulation.

Classifiers (rainfall)

PCA + logistic regression (q by leave-one-out CV), random forest (B = 5,000, m by out-of-bag error), and an rpart tree on 50 PLS components, as specified in the assignment.

Repeated cross-validation (rainfall)

10 repeats of stratified 5-fold CV on the 150 training stations, every tuning step of the corrected pipelines nested in the training fold, identical folds for every pipeline. Computed in R by scripts/a3_evaluation.R; details on the evaluation page.

Statistical helpers

Wilson intervals, normal and t quantiles, type-7 quantiles, exact binomial (McNemar) tests, the corrected resampled t interval and Holm's adjustment live in src/lib/stats and are unit-tested against values produced by base R (scripts/stats_reference.R).

Evaluation design

Unit of analysis: a weather station (150 for CV, 41 held back for the test set)
Resampling: 10 × stratified 5-fold CV; fold seeds 90138 + r; model seeds 90138 + 100r + k
Nesting: standardisation, PCA/PLS, q (LOOCV, 1–30) and m (OOB on 10 values) all refitted inside each training fold; the exception is the 2023-settings logistic pipeline, which keeps q = 27 chosen in 2023 on all 150 stations (a leak that can only favour it)
Accuracy intervals: Wilson 95% at n = 150 stations for CV (mean over repeats) and n = 41 for the test set
Paired differences: per-fold accuracy differences on identical folds; Nadeau–Bengio corrected resampled t interval (variance × (1/J + n_test/n_train))
Multiple comparisons: 9 paired tests in 3 families (method vs method; method vs baseline; corrected vs 2023); p-values Holm-adjusted within each family, intervals are per-comparison 95% intervals
Test-set comparisons: exact McNemar test on the discordant stations
Effect sizes: differences in accuracy in percentage points, with intervals; p-values are secondary
Uncertainty in PCA: bootstrap percentile intervals (resampling kernels), parallel analysis for the number of components

Headline

PCA + logistic regression has the highest point estimate, 94.7% cross-validated accuracy (95% CI 89.9–97.3%), but its advantage over the random forest, +2.4 pp (−1.5 pp to +6.2 pp), is not distinguishable from zero.

No self-scored metrics

Every number on the site is computed from the data by code in the repository, and the code is checked against R. Nothing is a self-assessment, and nothing claims external validation or certification.

Assumptions and limitations

Assumptions

Stations (and kernels) are treated as independent draws; spatial correlation between neighbouring stations is ignored.
The 150 training and 41 test stations come from the same population; the CV estimates describe performance on stations like these, not on Australia in general.
The Wilson interval at n = 150 treats the mean CV accuracy like a single binomial proportion; it captures sampling of stations but not the extra variability of refitting on different training sets.
Bootstrap percentile intervals assume the 210 kernels represent the population of kernels well enough for resampling to mimic new samples.

Limitations

41 test stations: one error moves test accuracy by 2.4 percentage points, so the test set cannot rank methods.
Repeated CV reuses the same 150 stations; consistent rankings across repeats are not independent evidence.
Eigenvector sign alignment cannot repair components that swap order between resamples; the stability score is there to flag it.
The random forests and the repeated CV run in R, not in your browser; the browser re-checks only what can be recomputed from published artefacts.
Derived rainfall artefacts still allow an approximate reconstruction of the standardised curves (DR-003).

What I'd change

Before analysing
Fix the analysis plan first: which matrix to decompose, which metric, which resampling design. Most of the 2023 bugs came from changing preprocessing mid-analysis.
While analysing
Lint R code (implicit assignment in if conditions), compute every projection with the same centring and scaling as the fit, and test preprocessing on held-out rows.
When reporting
Report intervals and paired differences by default, show tuning curves rather than just the argmin, and keep a decision record whenever an analysis choice is made.

Model card: rainfall classifiers

Rendered from docs/model-card.md. Its numbers are checked against the evaluation artefact by the test suite.

Three classifiers from MAST90138 Assignment 3 (University of Melbourne, 2023 Semester 2), re-run and re-evaluated in 2026. Each one labels an Australian weather station as North (0) or South (1) from its average rainfall on each of the 365 days of the year.

Model details

Aspect	PCA + logistic regression	Random forest	Tree on PLS components
Pipeline	standardise, PCA, logistic regression on the first q components	`randomForest`, B = 5,000 trees on all 365 days	standardise, kernel PLS (50 components), `rpart` tree on the scores
Tuning	q in 1..30 by leave-one-out CV (q = 3 on all 150 stations)	m by out-of-bag error on a 10-value grid (m = 57 on all 150 stations)	rpart defaults; the tree splits on C1 and C2
Software	R 4.6, `glm`; TypeScript port of `glm.fit` in the browser	R 4.6, randomForest 4.7-1.2	R 4.6, pls 2.9.0, rpart 4.1.27; TypeScript port in the browser
Seed	n/a (deterministic)	`set.seed(90138)`	`set.seed(90138)`

Author: Sunchuangyu (Rin) Huang. The 2023 versions contained bugs that are documented, not hidden: see the errata on the site and DR-001. This card describes the corrected pipelines unless stated otherwise.

Intended use

Teaching and portfolio demonstration of high-dimensional classification (p = 365 predictors, n = 150 stations): dimension reduction before classification, honest model comparison, and how small preprocessing slips change results.
Exploring, in the browser, how the choice of q, the PLS test scaling or the tuning loop affects the answer.

Out-of-scope uses

Operational climate or weather classification of real stations. The models were never validated outside this course data set and its time period.
Any decision about people. The data describe places; nothing here is suitable for decisions about individuals.

Training data

150 stations (58 North, 92 South), each a vector of 365 daily average rainfall values with a North/South label. Supplied with the subject; redistribution terms unknown, so the website publishes only derived quantities (DR-003).
Test data: 41 further stations (16 North, 25 South), used once per model.
Preprocessing: each day standardised with the training means and SDs (PCA and PLS pipelines); the random forest uses raw values.

Evaluation

Repeated stratified cross-validation on the 150 training stations (10 repeats x 5 folds, every tuning step of the corrected pipelines nested inside the training fold, identical folds for every pipeline), plus the 41-station test set. Accuracies are means over repeats with Wilson 95% intervals at n = 150; test accuracies have Wilson intervals at n = 41. Produced by scripts/a3_evaluation.R; intervals computed in web/src/lib/rainfall-evaluation.ts.

Pipeline	CV accuracy (95% CI)	Test accuracy (95% CI)	North recall in CV	South recall in CV
PCA + logistic regression	94.7% (89.9–97.3%)	40/41, 97.6% (87.4–99.6%)	91.6% (81.6–96.4%)	96.7% (90.8–98.9%)
Random forest	92.4% (87.0–95.7%)	38/41, 92.7% (80.6–97.5%)	87.8% (76.9–93.9%)	95.3% (88.9–98.1%)
Tree on PLS components	91.1% (85.5–94.7%)	40/41, 97.6% (87.4–99.6%)	92.1% (82.2–96.7%)	90.5% (82.8–95.0%)
Majority class (always South)	61.3% (53.3–68.8%)	25/41, 61.0% (45.7–74.3%)	0.0%	100.0%

Paired comparisons (Nadeau–Bengio corrected resampled t on the per-fold accuracy differences; nine tests in three families of three, so p-values are Holm-adjusted within each family):

Every method beats the majority baseline by about 30 percentage points (all corrected intervals exclude zero).
The three methods cannot be told apart: PCA + logistic minus random forest is +2.4 percentage points (95% CI −1.5 to +6.2), PCA + logistic minus PLS tree +3.6 (−1.8 to +9.1), random forest minus PLS tree +1.3 (−3.5 to +6.1). On the test set they disagree on at most two stations (exact McNemar p ≥ 0.5).
The 2023 bugs: fixing the PLS test scaling adds +23.6 percentage points (+9.6 to +37.6), Holm-adjusted p = 0.004, a clear effect. Fixing the logistic pipeline's settings points the same way, +11.1 (+0.8 to +21.3), but does not survive the adjustment for testing three fixes (Holm p = 0.069; on the test set the exact McNemar test is 3 stations to 0, p = 0.250), so it is suggestive rather than established. The 2023-settings logistic pipeline keeps q = 27, which was chosen in 2023 on all 150 stations, so its CV accuracy carries a small leak that can only favour it; the gain from the fix is, if anything, understated. Fixing the random forest's m loop changes nothing (−0.1, −2.4 to +2.2).

The random forest's own out-of-bag error (6.7%, 95% CI 3.7–11.8%) agrees with its repeated-CV error (7.6%, 4.3–13.0%) and its test error (7.3%, 2.5–19.4%).

Known failure modes

Northern stations are the harder class for the logistic model and the forest (recall about 88–92% against 95–97% for the South). North is also the smaller class (58 of 150).
Preprocessing mismatches are catastrophic. Projecting unstandardised test rows onto PLS directions learned from standardised data (the 2023 slip) drops CV accuracy to 67.6% and South recall to about 52%. Any new data must be standardised with the training means and SDs.
Tuning is weakly identified. Inside the 50 outer folds, q ranged from 2 to 9 and m from 5 to 153; the OOB error curve is nearly flat. The specific values q = 3 and m = 57 should not be read as meaningful optima.
Small test set. With 41 stations, a single error moves test accuracy by 2.4 percentage points; test-set intervals are 12 to 29 points wide.
Distribution shift. The curves are long-run averages for one period. Stations in transition zones, different years, or a changing climate may not look like the training data.

Ethical considerations

No personal data: the observations are weather stations. The main data concern is licensing of the course-supplied files, handled by publishing derived quantities only. A disclosure check shows the published PLS projections still allow an approximate reconstruction of the standardised training curves; this is recorded in DR-003 as an open decision.
Low stakes, high temptation to over-claim: a 41-station test set can make a model look perfect. This card reports cross-validated intervals and paired comparisons so the methods are not ranked on noise.
The optional AI feature on the site does not touch these models or their data.

Caveats and recommendations

Treat the three pipelines as equally good on this data; prefer the simplest (PCA + logistic regression on a few components) if one must be chosen, and say that the choice is not statistically supported.
Re-estimate with fresh stations before drawing conclusions about other regions or periods.

Data card: wheat kernels (Assignment 1)

Source: UCI Machine Learning Repository "seeds" (Charytanowicz et al., 2010), CC BY 4.0. 210 kernels, 70 each of Kama, Rosa and Canadian wheat, 7 geometric measurements from soft X-ray images.
Use on the site: principal component analysis in the browser, with bootstrap intervals and parallel analysis (DR-002). No model is trained to predict variety.
Limitations: one harvest and one laboratory; measurements on very different scales (mm² to dimensionless), which drives the covariance PCA.

Decision records

Rendered from docs/decisions/. Each states the decision first, then the context, the options considered, why, what happened (weak numbers included) and what I would change. Accepted records are never edited; a new record supersedes an old one.

DR-001 The m_OOB bug in the random forest tuning loop
DR-002 Scaled versus unscaled PCA for the wheat data
DR-003 Not publishing the raw rainfall data

DR-001 · Accepted · 2026-10-06

The m_OOB bug in the random forest tuning loop

Applies to: Assignment 3, Question 3(1); /rainfall#q3, /rainfall/evaluation

Decision in one line: keep the 2023 result exactly as submitted, show a corrected re-run next to it, and report the number of split candidates m as weakly identified instead of claiming an optimal value.

Context

Assignment 3 asked for a random forest of B = 5,000 trees on all 365 daily rainfall values, with the number of variables tried at each split (m) chosen by out-of-bag (OOB) error on a ten-value grid around c = √365 ≈ 19.1 (c/4, c/2, c, 2c, ..., 8c). My 2023 loop compared the OOB errors with

if (oob_error <- min_oob_error) {

which assigns Inf to oob_error instead of comparing. Inf is truthy, so every candidate "won" and best_m always ended as the last grid value, 8c ≈ 152.84 (used as mtry 153). The grid was also rounded to four decimals rather than to integers, which turned out to be harmless because randomForest rounds mtry itself. The report gave 3 test errors out of 41.

Decision

Keep the as-submitted result visible and unchanged, add a corrected re-run (strict <, integer grid, smallest m on ties) behind an As submitted / Corrected switch, and measure whether the bug changed any conclusion before saying anything about which m is best.

Options considered

Fix it silently and show only the corrected m. Cleaner, but it rewrites the record of what I actually submitted.
Show only the original. Faithful, but leaves a known bug unexplained.
Show both and quantify the difference (chosen): re-run the corrected loop with the original seed, then re-estimate both versions with repeated cross-validation on identical folds.

Why

The original submission is a record and stays faithful; the upgrade adds analysis around it. A bug that changes the chosen m is only worth fixing loudly if it changes what the model does, and that is an empirical question with a measurable answer.

What happened

The corrected loop picks m = 57 with the lowest OOB error, 6.67% (10 of 150 stations). The whole OOB curve over the ten grid values runs from 6.67% to 8.00%, a difference of two stations.
Both versions make the same 3 errors on the 41 test stations (7.3%, Wilson 95% CI 2.5–19.4%).
Repeated stratified cross-validation (10 repeats x 5 folds, m tuned inside every training fold): corrected forest 92.4% accuracy (95% CI 87.0–95.7%), forest with the 2023 m 92.5% (87.2–95.8%). The paired difference is −0.1 percentage points (corrected 95% interval −2.4 to +2.2). The bug made no measurable difference.
When each of the 50 outer folds tuned m on its own training data, it chose values from 5 to 153; the smallest grid value (5) won in 17 of 50 folds and 8 of the 10 grid values won at least once. Within a fold the OOB error varied by only 0.8–3.3 percentage points (one to four stations) across the whole grid. So "m = 57" is not a meaningful optimum; the OOB curve is flat.
The OOB error itself is trustworthy here: per fold, OOB minus held-out error averages −0.6 percentage points (corrected 95% interval −5.8 to +4.5).

The weak number to be honest about: correcting the bug improved nothing. Its value is in the record, not in the accuracy.

What I'd change

Catch it mechanically: lintr::implicit_assignment_linter() flags assignments inside if () conditions.
Treat m as a low-sensitivity parameter: use the default √p, or a one-standard-error rule on the OOB curve, and report the OOB error with its Monte Carlo spread across seeds rather than a single minimum.
Report the tuning curve, not just the argmin, so a flat curve is visible at a glance.

DR-002 · Accepted · 2026-10-06

Scaled versus unscaled PCA for the wheat data

Applies to: Assignment 1, Problem 2; /seeds#pca, /seeds#uncertainty

Decision in one line: present the covariance-matrix PCA (what the question asked) as the primary analysis, keep the correlation-matrix PCA as a clearly labelled alternative, keep the 2023 mixed version visible as errata, and put bootstrap intervals and parallel analysis around both.

Context

Assignment 1 Problem 2 asked for a principal component analysis of the sample covariance matrix of seven kernel measurements (210 kernels, three varieties). The 2023 submission ran prcomp(X, scale = TRUE), which is a PCA of the correlation matrix, for the eigenvalues, scree plot and correlation circle. The PC1/PC2 scatter was then drawn as as.matrix(X) %*% gamma: raw, uncentred measurements multiplied by loadings estimated for standardised variables. The submission concluded that three components should be kept.

The variables are on different scales: area in mm², lengths in mm, compactness dimensionless, asymmetry a coefficient. Area has by far the largest variance.

Decision

Show all three versions behind a switch, with the covariance PCA selected by default because it answers the question that was asked. Label the correlation PCA as the analysis I would choose for variables in different units, and keep the as-submitted mixture as the record of what was handed in.

Options considered

Covariance PCA only. Answers the question, but hides why standardising matters.
Correlation PCA only. Arguably better practice for mixed units, but it is not what was asked and it would quietly endorse half of the 2023 mix-up.
All three, labelled (chosen), plus uncertainty for the two consistent analyses.

Why

Faithfulness: the original answer stays visible and its two analyses are separated rather than overwritten. Teaching value: switching between the covariance and correlation PCA shows directly how a variable's scale drives the first component. Rigour: a point estimate of "variance explained" from one sample of 210 kernels says nothing about how stable it is.

What happened

Bootstrap (1,000 resamples of kernels, seed 90138, whole PCA redone each time, 95% percentile intervals):

Covariance PCA. PC1 explains 82.9% of the variance (95% CI 79.8–85.9%) and is essentially area (loading 0.884, CI 0.873–0.891) and perimeter. PC2 (16.4%, CI 13.5–19.5%) is almost entirely asymmetry (loading 0.989). Every eigenvector is very stable (median |cosine| with the full-sample direction at least 0.994; PC5 is the least stable).
Correlation PCA. PC1 explains 71.9% (69.9–74.1%), PC2 17.1% (15.0–19.2%), PC3 9.7% (7.8–11.5%); three components keep 98.7% (98.4–98.9%), the 2023 figure.
How many components? The rules disagree. The cumulative-variance rule (ψ ≥ 95%) gives 2 for the covariance PCA and 3 for the correlation PCA; Kaiser's rule gives 2 for both; parallel analysis with a column-permutation null (1,000 permutations) gives 1 for both at the 95th percentile, and 1 (covariance) or 2 (correlation) at the null mean.

The weak point to be honest about: the 2023 "keep three components" rests on the variance rule alone. By parallel analysis only the first component is clearly more than chance, and covariance PC2 is one variable's variance (asymmetry) rather than shared structure. That does not make PC2 useless: it is what separates Canadian from Kama kernels in the scatter. Parallel analysis tests for correlation structure, not for usefulness in telling varieties apart.

What I'd change

Decide which matrix to decompose before running anything, and say why in one sentence (units differ, so standardise, unless the question fixes it).
Never project data with loadings from a different preprocessing; compute scores with the same centring and scaling as the eigen-decomposition.
Report variance shares with intervals and use more than one rule to choose the number of components, stating which question each rule answers.

DR-003 · Accepted (two follow-ups open) · 2026-10-06

Not publishing the raw rainfall data

Applies to: Assignment 3 data; web/public/data/, coursework/assignment-3/

Decision in one line: the website serves only derived quantities from the Assignment 3 rainfall data, never the raw station rows; the raw files stay in the (private) repository so the analysis can be reproduced and tested.

Context

Assignment 3 used daily average rainfall for 191 Australian weather stations (150 training, 41 test), each a curve of 365 values with a North/South label. The files were supplied with the subject and I do not know their redistribution terms. The revived site needs station-level results to be interactive (PC scores for the logistic model, PLS scores for the tree, test-set predictions), and the new evaluation needs per-fold results.

Decision

Publish only what is derived from the data: class-level daily summaries (means and quantiles), principal component and PLS scores, importances, fitted trees, confusion counts, per-fold evaluation summaries and test-set predictions. Keep XGtrainRain.txt and XGtestRain.txt in coursework/assignment-3/, read only by the R scripts and by the test suite's parity checks, and never bundle them into the site.

Options considered

Publish the raw data on the site. The simplest, and it would let every model be refitted in the browser, but it redistributes course material whose licence is unknown.
Publish nothing station-level (static figures only). Safe, but it would remove the interactive labs and any way to check the numbers.
Derived quantities only (chosen), with parity tests that read the raw files at test time and prove the published numbers come from them.

Why

When the licence is unclear, the conservative default is not to redistribute. Derived projections keep the labs interactive, and the parity tests keep them honest without shipping the source rows.

What happened

The labs work from the derived files (about 440 KB in web/public/data/). Two things cannot run in the browser and are computed in R instead: the random forests and the repeated cross-validation. They are shipped as per-forest and per-fold summaries.
"Derived" is not the same as "irreversible". A disclosure check in the test suite (rainfall-disclosure.test.ts) shows that the published 50 PLS scores, together with the published correlation of every day with every PLS component, reconstruct about 96.6% of the variance of the standardised training curves (median per-station correlation 0.91). Weather stations are places, not people, so this is not a privacy risk; it does mean the site republishes most of the information in the training curves in a transformed form.
The raw files are committed in the repository, which is private. Making the repository public would publish them.

These two points are open owner decisions, not things to settle silently in code.

What I'd change

Before making the repository public: ask the subject coordinator whether the files may be redistributed, or replace them with a download script for equivalent public Bureau of Meteorology data.
Publish the day-by-component correlations only for the seven components the PLS lab displays (C1 to C3, C5 to C7 and C14, not all 50), round the PLS scores more coarsely, and re-run the disclosure check to confirm how much that reduces what can be reconstructed.
Run the disclosure check whenever a new derived artefact is added, not just once.

AI use statement

Informed by the Australian Government's policy for the responsible use of AI in government (DTA), the transparency principles of the EU AI Act and the NIST AI Risk Management Framework. This is a personal portfolio site: it is not assessed against, and does not claim compliance with, any of them.

What the AI does

One optional feature: “Explain this projection” on the Seeds PCA page writes a short plain-language reading of the PC scatter you are looking at.
It runs only if you add your own API key (Anthropic by default, model Claude Haiku 4.5, or Claude Sonnet 5.5; or OpenAI, default gpt-5-mini).
The reply must match a fixed JSON schema and is validated before it is shown.

What it never does

It never computes or changes any result, chart or table on the site.
It never sees kernel-level or station-level data, and never the rainfall data.
It never runs without your key and your click; the site works fully without it.
Its output is never shown without the “AI-generated” label.

Data sent to the provider

Fixed instructions plus one JSON summary, shown in full for every call in the audit log. The summary contains:

a one-line dataset description (210 kernels, the number per variety, seven measurements);
the analysis variant and, for the 2023 version, a fixed note on how its scores were computed;
for the two plotted components: eigenvalue, variance share with its bootstrap interval, and the loadings of all seven variables;
the cumulative variance shares of all seven components;
the bootstrap settings (seed and number of resamples) and the parallel-analysis count;
each variety's mean and SD on the two plotted components;
two fixed notes on the intervals and on eigenvector signs.

No kernel-level row is sent. Your key goes from your browser directly to the provider in a request header; it is never sent to this site, never logged and never stored in the audit log.

Human in the loop and audit

Each explanation stays marked “pending” until you accept, edit or reject it, and that decision is recorded; a rejected explanation is hidden from the page. A grounding check lists any number in the reply (and, after an edit, in your version) that does not appear in the summary that was sent. Every call (including failures and cancellations) is written to an audit log in your browser with its time, provider, model, input, output, latency, token usage and your decision; view or export it as JSON or CSV on the AI audit log page. If the browser blocks IndexedDB, the log is kept for the open tab only and the page says so.