Turn alternative data into an auditable project

Learning goals

• How many usable rows remain?
• Explain why when the information was actually available applies to this case.
• Identify the case-specific assumption or failure mode before taking action.

Draft · approximately 50 minutes, excluding further revision.

Shared case and decision context

A research team receives 1,000 transaction rows: 80 duplicates and 20 additional unusable rows after deduplication. Fields include merchant identifiers and dates. The provider prohibits re-identification. A vendor revises historical records without a version log.

Economic or professional mechanism

A data project begins with a decision and target, followed by permission, provenance, cleaning, transformation, modeling and evaluation. Volume does not cure selection bias: card users or covered merchants may not represent the investable company’s customers. Timestamp alignment must distinguish transaction time, publication time and the moment an investor could have known the value.

Apply, challenge and communicate

The usable count is 900. Archive lawful point-in-time versions with a reproducible cleaning log and access controls. Do not combine identifiers to bypass the contractual privacy restriction. Evaluate text or transaction features on unseen periods and document revision policy; a backtest using revised history may overstate implementable results.

Explore before fitting

Inspect distributions, missingness, outliers, time coverage, correlations and target imbalance before model fitting. The objective is to understand the dataset and detect failures, not to repeatedly search the future test sample for profitable patterns. Exploration can uncover provider revisions and concentration in certain merchants, requiring redesign of the sampling or feature plan. Document each material choice so a later analyst can reproduce the pipeline.

Represent text without leaking labels

Text features can use token counts, term-frequency/inverse-document-frequency weights, sentiment measures or learned embeddings. Cleaning, vocabulary fitting and feature selection must be learned within training data. Negation, language, domain-specific meanings and timestamp availability matter in financial text. A phrase appearing after an earnings release cannot be used to forecast that same release as if it were available earlier.

Separate training, tuning and final evaluation

Training estimates model parameters; validation chooses hyperparameters or decision thresholds; a final untouched test estimates performance after those choices. Cross-validation must respect the time and dependence structure. Standardization, imputation and feature selection belong inside each training fold. Reusing the final test for every experiment turns it into another validation sample and invalidates the claimed independent assessment.

Audit a financial-text forecasting pipeline

A reproducible text pipeline records document version, publication timestamp, source permission, language detection, deduplication and tokenization before feature construction. Align the forecast horizon to when the text could be received and processed, including operational delay. Explore whether duplicated articles or multiple syndicated copies inflate apparent sample size. Evaluate unseen periods and preserve access controls for any sensitive text.

Formula and assumptions

usable rows=raw−duplicates−additional unusable rows

Case analysis with explicit assumptions

1. A research team receives 1,000 transaction rows: 80 duplicates and 20 additional unusable rows after deduplication. Fields include merchant identifiers and dates. The provider prohibits re-identification. A vendor revises historical records without a version log.
2. Method: usable rows=raw−duplicates−additional unusable rows
3. Case result: 900 rows. The usable count is 900. Archive lawful point-in-time versions with a reproducible cleaning log and access controls. Do not combine identifiers to bypass the contractual privacy restriction. Evaluate text or transaction features on unseen periods and document revision policy; a backtest using revised history may overstate implementable results.

Errors to catch

• Do not replace the stated assumptions with an unstated shortcut. Reconcile units, timing and the claim being valued or duty being assessed.
• All selection bias automatically is not a defensible conclusion for these case facts.

Original knowledge check

Answer every question before checking. Review explanations after submitting; retry starts a new attempt.

Sources and coverage status

These original lessons are drafts. Public outlines are used to check scope; the official learning-outcome text and proprietary question banks are not reproduced here. Follow the source for exam-year requirements and updates.

• Reference · www.cfainstitute.org
• Reference · www.itl.nist.gov