This page reviews "...and the Cross-Section of Expected Returns," a 2016 paper by Campbell Harvey, Yan Liu, and Heqing Zhu. The researchers argued that a significant portion of the hundreds of stock market patterns ("factors") published in academic journals may be false discoveries, the result of data mining rather than genuine economic relationships. Their paper raised the statistical bar for what counts as a real factor and called into question a large portion of the factor investing literature.

Published in The Review of Financial Studies, the paper documented that researchers had published at least 316 factors claimed to predict stock returns. The authors showed that the standard statistical test used to validate these discoveries (a t-statistic above 2.0) was far too lenient given how many factors had been tested. After adjusting for the sheer volume of testing, the paper concluded that the real threshold for a credible new factor should be roughly 3.0, and that many published factors would fail this higher bar.

Key Findings

The paper addresses a fundamental problem in empirical research: when thousands of potential patterns are tested on the same data, some will appear statistically significant purely by chance. This is the multiple testing problem, and the authors argued that academic finance had largely ignored it.

The Factor Zoo

The researchers catalogued 316 factors published in top academic journals between 1967 and 2014. These factors cover an enormous range of stock characteristics: accounting ratios, trading patterns, corporate events, macroeconomic sensitivities, and more. Each paper claimed to have found a new variable that predicts which stocks will earn higher or lower returns.

The rate of new factor discoveries accelerated over time. In the 1970s and 1980s, a handful of new factors were published per year. By the 2010s, the rate had climbed to over 40 per year. The authors argued that this acceleration was not driven by genuine new discoveries about how markets work; it was driven by publication incentives that reward novel findings and by the availability of the same widely-used financial databases that every researcher mines for patterns.

The Multiple Testing Problem

The standard approach in academic finance is to consider a factor "statistically significant" if its t-statistic exceeds 2.0. This threshold corresponds to roughly a 5% chance that the result occurred by random chance (a p-value of 0.05). If you test only one factor, a 5% false positive rate is reasonable.

The problem is that hundreds of researchers, across decades, have been testing hundreds of potential factors on the same or overlapping datasets. When 316 factors are tested, even if none of them are real, you would expect about 16 to appear significant at the 5% level purely by chance. The standard threshold does not account for this accumulated testing.

The researchers applied multiple testing corrections (specifically, the Bonferroni correction and methods based on controlling the false discovery rate) to calculate what the t-statistic threshold should be given the number of factors already tested. Their conclusion: by 2012, a new factor needed a t-statistic of at least 3.0 to be considered credible, and by 2032 (accounting for continued testing), the threshold would need to be even higher.

Which Factors Survive?

When the researchers applied the higher threshold to the 316 published factors, many did not survive. The paper does not provide a definitive list of "real" versus "false" factors, but the implication is clear: a substantial fraction of published factors are likely the product of data mining.

The factors most likely to survive the higher bar are those with the strongest statistical evidence, clear economic rationale, and robustness across different time periods and markets. The original Fama-French factors (market, size, value) and momentum have t-statistics well above 3.0 in most studies. Many of the more exotic or niche factors do not.

Practical Implications

Choosing Which Factors to Invest In

The paper has direct implications for anyone building a factor-based investment strategy. If many published factors are false discoveries, then building a portfolio around a large number of factors is risky: some of those factors will not produce the expected returns going forward because they never reflected a real pattern in the first place.

The practical takeaway is to focus on factors with the strongest evidence: high t-statistics, economic intuition for why the premium should exist, evidence across multiple countries and time periods, and robustness to different definitions and measurement methods. A handful of well-established factors (market, value, momentum, profitability, investment) have cleared these bars. Hundreds of others have not.

Healthy Skepticism of Backtests

The paper reinforces a broader point about backtesting: any strategy that looks good in historical data might be the result of data mining rather than a genuine edge. The more strategies you test, the more likely you are to find one that appears to work by chance.

For investors evaluating quantitative strategies, this means asking hard questions. How many variations did the developer test before arriving at this one? Was the strategy tested on out-of-sample data (data that was not used to develop the strategy)? Does the strategy have a clear economic explanation, or is it just a pattern that happened to appear in historical data?

Publication Bias in Finance Research

The paper highlights a structural problem in academic publishing. Journals strongly prefer to publish papers that find statistically significant results. Researchers who test a factor and find nothing interesting are unlikely to publish that null result. This creates survivorship bias in the published literature: we see the factors that appeared significant but not the many more tests that found nothing.

The consequence is that the published record of factor discoveries overstates how many real factors exist. For every factor that appears in a journal, there may be dozens of unpublished tests that found no effect. This "file drawer problem" means the published t-statistics are biased upward, making the multiple testing adjustment even more important.

How the Researchers Tested This

Building the Factor Catalogue

The researchers systematically reviewed papers published in top finance and economics journals (The Journal of Finance, Journal of Financial Economics, Review of Financial Studies, and others) and catalogued every factor claimed to predict the cross-section of stock returns. For each factor, they recorded the t-statistic, the publication year, the journal, and other details.

This catalogue provides a unique bird's-eye view of the factor literature. It shows how the rate of discovery has accelerated, how t-statistics have evolved over time, and how the collective body of research relates to the multiple testing problem.

The Statistical Framework

The paper uses three approaches to adjust for multiple testing. The Bonferroni correction is the simplest and most conservative: it divides the significance threshold by the number of tests, requiring much stronger evidence for each individual test. The Holm correction is slightly less conservative and accounts for the ordering of test statistics. The Benjamini, Hochberg, and Yekutieli (BHY) method controls the false discovery rate (the expected proportion of false positives among discoveries) rather than the probability of any false positive.

All three methods point in the same direction: the standard t-statistic threshold of 2.0 is too low given the volume of testing that has occurred. The BHY method, which is the least conservative of the three, still requires a t-statistic well above 2.0 for factors tested after the early 2000s.

A Threshold That Rises Over Time

A distinctive feature of the paper is that the required t-statistic threshold increases over time. As more factors are tested, the multiple testing correction becomes more severe. A factor published in 1970, when few factors had been tested, needed a lower t-statistic than a factor published in 2010, when hundreds of factors had already been examined.

The researchers projected that by the mid-2030s, a t-statistic of approximately 3.4 would be needed for a new factor to be considered credible. This rising bar means that finding genuinely new factors becomes progressively harder, not because markets have changed, but because so much testing has already been done on the available data.

Limitations and Caveats

Related Research

Further Reading

• Harvey, C.R., Liu, Y., and Zhu, H. (2016). "...and the Cross-Section of Expected Returns." The Review of Financial Studies, 29(1), 5–68.
• McLean, R.D. and Pontiff, J. (2016). "Does Academic Research Destroy Stock Return Predictability?" The Journal of Finance, 71(1), 5–32.
• Hou, K., Xue, C., and Zhang, L. (2020). "Replicating Anomalies." The Review of Financial Studies, 33(5), 2019–2133.
• Linnainmaa, J.T. and Roberts, M.R. (2018). "The History of the Cross-Section of Stock Returns." The Review of Financial Studies, 31(7), 2606–2649.
• Harvey, C.R. (2017). "Presidential Address: The Scientific Outlook in Financial Economics." The Journal of Finance, 72(4), 1399–1440.
• Benjamini, Y. and Hochberg, Y. (1995). "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing." Journal of the Royal Statistical Society: Series B, 57(1), 289–300.