Learn signals without leaking the answer

Learning goals

• Calculate precision for positive default alerts.
• Explain why 75% applies to this case.
• Identify the case-specific assumption or failure mode before taking action.

Draft · approximately 45 minutes, excluding further revision.

Shared case and decision context

A credit classifier labels 100 borrowers: TP=18, FP=12, FN=6, TN=64. Defaults are the positive class. Tuning used only a training/validation split, and a final untouched chronological test remains available.

Economic or professional mechanism

Supervised learning uses labeled outcomes; unsupervised learning identifies structure without a target label. Classification metrics answer different economic questions: precision measures the fraction of positive alerts that are true, while recall measures detected defaults among all actual defaults. Accuracy can look attractive in an imbalanced dataset while missing costly defaults.

Apply, challenge and communicate

Precision is 60% and recall is 75%. Choose thresholds using false-positive and false-negative costs, not a cosmetic score. Fit preprocessing only within training folds; otherwise scaling, imputation or feature selection can leak information. Preserve the untouched test for a final estimate, then monitor drift after deployment.

Build a validation chronology that survives implementation

A training set fits model parameters; a validation set chooses hyperparameters; a final untouched test set estimates performance after those choices. In financial series use a chronology consistent with the information available on each historical decision date. Scaling, missing-value imputation, feature selection and resampling must be fitted within training folds, not on the full sample.

For a six-month forward-return label, adjacent observations have overlapping outcome windows. A nominal date split can therefore share future return information across folds. Purge training observations whose labels overlap the validation period and consider a suitable embargo for the process being modeled; its length must follow the label construction, not a universal setting. Repeatedly choosing the best model on the final test set turns it into another validation set. Report economic costs, turnover, capacity, calibration and stability alongside accuracy.

Formula and assumptions

precision=TP/(TP+FP); recall=TP/(TP+FN)

Case analysis with explicit assumptions

1. A credit classifier labels 100 borrowers: TP=18, FP=12, FN=6, TN=64. Defaults are the positive class. Tuning used only a training/validation split, and a final untouched chronological test remains available.
2. Method: precision=TP/(TP+FP); recall=TP/(TP+FN)
3. Case result: 60% precision. Precision is 60% and recall is 75%. Choose thresholds using false-positive and false-negative costs, not a cosmetic score. Fit preprocessing only within training folds; otherwise scaling, imputation or feature selection can leak information. Preserve the untouched test for a final estimate, then monitor drift after deployment.

Errors to catch

• Do not replace the stated assumptions with an unstated shortcut. Reconcile units, timing and the claim being valued or duty being assessed.
• Only the largest reported accuracy is not a defensible conclusion for these case facts.

Original knowledge check

Answer every question before checking. Review explanations after submitting; retry starts a new attempt.

Sources and coverage status

These original lessons are drafts. Public outlines are used to check scope; the official learning-outcome text and proprietary question banks are not reproduced here. Follow the source for exam-year requirements and updates.

• Reference · www.cfainstitute.org
• Reference · www.itl.nist.gov