Estimate carefully and test a stated claim

Learning goals

• Build a confidence interval while identifying sampling limitations.
• Interpret significance, power and the choice between parametric and rank-based tests.

Draft · approximately 60 minutes, excluding further revision.

A sample is not the population

A random sample gives each eligible observation a defined selection mechanism. Stratification samples within groups; cluster sampling selects whole groups; convenience sampling may leave a large selection bias regardless of sample size. Time-series observations can be dependent, and excluding failed funds creates survivorship bias. Increasing n does not repair a systematically biased sample.

For independent observations with finite variance, the central limit theorem motivates an approximately normal distribution of the sample mean as n grows; it does not make raw returns normal. Estimated standard error is s/√n. With unknown population variance and appropriate assumptions, a mean interval is x̄ ± t* s/√n. Interpret a 95% confidence method by its long-run coverage, not a claim that 95% of observations lie inside this mean interval.

Set hypotheses before seeing the result

State H₀, H₁, a one- or two-sided alternative and significance α before searching the data. For a mean, t = (x̄−μ₀)/(s/√n). Reject when the p-value is at or below the chosen α. The p-value is the probability, under H₀ and model assumptions, of a statistic at least as extreme as observed; it is not the probability that H₀ is true. Statistical significance does not establish economic profitability after costs.

Type I error rejects a true H₀; Type II error fails to reject a false H₀; power = 1−β. Lowering α with fixed data can reduce power. Paired data require a test on matched differences rather than an independent-samples formula. A sign test uses the binomial count of positive differences under a 0.5 null; rank procedures can reduce sensitivity to distribution assumptions but still require conditions such as independent pairs. They are not assumption-free substitutes.

Separate uncertainty in a mean from variability of returns

A hypothetical independent sample of 64 monthly excess returns has mean 0.5% and sample standard deviation 4%. Standard error is 4%/sqrt(64)=0.5%. With a supplied two-sided critical t value of 2.00, a mean confidence interval is −0.5% to 1.5%. This interval concerns the population mean under the stated assumptions; it is not a forecast interval for the next monthly return.

At a fixed confidence level and unchanged variability, halving the standard error needs about four times as many independent observations. Simply collecting overlapping returns does not necessarily supply that effective sample size. Report the estimate, uncertainty, sampling method and economic magnitude together. A small p-value cannot repair look-ahead bias or compensate for an unreported search across many strategies.

Formula and assumptions

SE(x̄) = s/√n; t = (x̄−μ₀)/SE; CI = x̄ ± t*SE; power = 1−β

A mean interval and a small sign test

1. For n=25, mean return 1%, sample standard deviation 5%: standard error = 5%/5 = 1%. Using supplied t*=2.064, 95% interval is −1.064% to 3.064%.
2. Testing H₀: mean = 0 gives t=1. With a supplied two-sided critical value 2.064, do not reject H₀. This does not prove the mean equals zero.
3. For eight independent nonzero paired differences, all positive, the two-sided sign-test p-value is 2×(0.5)⁸ = 0.0078125. Ties must be excluded or otherwise handled under a stated convention.

Errors to catch

• Reading “do not reject” as proof of equality.
• Choosing the test direction after seeing the sign of the data.

Original knowledge check

Answer every question before checking. Review explanations after submitting; retry starts a new attempt.

Sources and coverage status

These original lessons are drafts. Public outlines are used to check scope; the official learning-outcome text and proprietary question banks are not reproduced here. Follow the source for exam-year requirements and updates.

• Reference · www.cfainstitute.org
• Reference · www.itl.nist.gov