From a dataset to a defensible model

Learning goals

• Identify suitable roles for machine learning and alternative financial data.
• Design a workflow that prevents leakage and documents limits.

Draft · approximately 60 minutes, excluding further revision.

The research question comes before the algorithm

Financial data science combines a measurable question, reliable point-in-time data, a model and an evaluation design. Structured tables and unstructured text can both contain useful information. Large volume does not establish truth, permission to use data, representativeness or relevance to an investment horizon. Record the original source, collection time, publication lag and permitted use before engineering a feature.

Supervised learning uses labeled outcomes, such as a default flag or a future return. Unsupervised learning looks for structure without those target labels, such as clusters of similar businesses. Deep learning is a family of multi-layer models; it is not a guarantee of superiority. A language model can generate plausible unsupported text, so it cannot replace source verification or a controlled valuation process.

Test the decision that could actually have been made

Divide data into training, validation and untouched testing periods with time order respected. Fit scaling, imputation and feature selection on training data only, then apply that transformation to later data. A random time-series split can allow future regimes or revised accounting data to leak into a supposed historical decision. Keep a simple baseline and account for fees, turnover and class imbalance in evaluation.

Overfitting means a model learns sample-specific noise rather than a relationship that survives new observations. Regularization, simpler models and disciplined validation can help but do not guarantee stability. Monitor drift after deployment, document human review and provide a way to stop using a model. This lesson is conceptual; IBBC Capital is not running an automated trading model or an AI tutor through this page.

A point-in-time dataset is part of the investment method

A year-end financial metric becomes public on 15 March. A strategy trading on 31 January cannot use that metric merely because its database labels it with the preceding 31 December accounting date. Store period-end, publication timestamp and revision timestamp separately. In an original research notebook, specify the earliest tradable date after release and any execution lag.

Split a time series chronologically before fitting preprocessing steps. Estimate missing-value replacements and scaling parameters on training data only, then apply them unchanged to validation data. Tuning on the test set converts it into another training input. A final holdout should remain unused until the method is fixed; report turnover, costs, drawdowns and stability, not only classification accuracy.

Formula and assumptions

precision = TP/(TP+FP); recall = TP/(TP+FN); evaluation must respect information availability

A forecast that accidentally sees the future

1. A researcher predicts next month’s default using a financial ratio whose final restatement was published six months later. The historical table contains the revised value but no availability timestamp.
2. Rebuild the feature from information actually published by the decision date. If that point-in-time evidence is unavailable, exclude or clearly restrict the feature rather than assigning a favorable backtest result.
3. Use chronological validation and an untouched later period; compare default-detection precision and recall with a simple baseline. A 99% accuracy can be achieved by always predicting “no default” when only 1% default, while detecting no defaults at all.

Errors to catch

• Using revised data as though it had been available at the original date.
• Calling high accuracy proof of useful performance in a rare-event dataset.

Original knowledge check

Answer every question before checking. Review explanations after submitting; retry starts a new attempt.

Sources and coverage status

These original lessons are drafts. Public outlines are used to check scope; the official learning-outcome text and proprietary question banks are not reproduced here. Follow the source for exam-year requirements and updates.

• Reference · www.cfainstitute.org