Cleaning large correlation matrices: Tools from Random Matrix Theory
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
This review covers recent results concerning the estimation of large covariance matrices using tools from Random Matrix Theory (RMT). We introduce several RMT methods and analytical techniques, such as the Replica formalism and Free Probability, with an emphasis on the Marčenko–Pastur equation that provides information on the resolvent of multiplicatively corrupted noisy matrices. Special care is devoted to the statistics of the eigenvectors of the empirical correlation matrix, which turn out to be crucial for many applications. We show in particular how these results can be used to build consistent “Rotationally Invariant” estimators (RIE) for large correlation matrices when there is no prior on the structure of the underlying process. The last part of this review is dedicated to some real-world applications within financial markets as a case in point. We establish empirically the efficacy of the RIE framework, which is found to be superior in this case to all previously proposed methods. The case of additively (rather than multiplicatively) corrupted noisy matrices is also dealt with in a special Appendix. Several open problems and interesting technical developments are discussed throughout the paper.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Common risk factors in the returns on stocks and bonds
Fama, E. F., & French, K. R. (1993) · Journal of Financial Economics · doi:10.1016/0304-405x(93)90023-5 · Cited by 25,754
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
McCullagh, P., & Nelder, J. A. (1989) · doi:10.1007/978-1-4899-3242-6 · Cited by 16,836
Large Sample Properties of Generalized Method of Moments Estimators
Hansen, L. P. (1982) · Econometrica · doi:10.2307/1912775 · Cited by 13,467
A planar diagram theory for strong interactions
Hooft, G. '. (1974) · Nuclear Physics B · doi:10.1016/0550-3213(74)90154-0 · Cited by 4,476 · cited 1×
“This concept of asymptotic freeness is also related to the notion of vanishing non-planar diagrams [80].”
BackgroundSpin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications
Mezard, M., Parisi, G., & Virasoro, M. (1986) · World Scientific Lecture Notes in Physics · doi:10.1142/0271 · Cited by 3,653
A well-conditioned estimator for large-dimensional covariance matrices
Ledoit, O., & Wolf, M. (2004) · Journal of Multivariate Analysis · doi:10.1016/s0047-259x(03)00096-4 · Cited by 3,046 · cited 10×
“The first step is to estimate the population eigenvalues using the QuEST method of Ledoit and Wolf (see Section 8.1.3).”
BackgroundInfluentialQuantum statistical mechanics in a closed system
Deutsch, J. M. (1991) · Physical Review A · doi:10.1103/physreva.43.2046 · Cited by 2,860 · cited 2×
“where we used again an “ergodic hypothesis” [127] as N →∞ in the last step.”
BackgroundRandom-matrix theory of quantum transport
Beenakker, C. W. J. (1997) · Reviews of Modern Physics · doi:10.1103/revmodphys.69.731 · Cited by 2,551 · cited 1×
“In this appendix, we consider the case of an additive external noise which can also be important in many situations, in particular in quantum chaos and quantum transport [185], with renewed interest coming fromproblems of quantum ergodicity (‘‘eigenstate thermalization’’) [130,186], entanglement and dissipation (for r…”
BackgroundBrézin, E., Itzykson, C., Parisi, G., & Zuber, J. B. (1978) · Communications in Mathematical Physics · doi:10.1007/bf01614153 · Cited by 1,479 · cited 5×
“We refer to [52] for the moment methods (ii), to [41, 66] for Feynman diagrams (iii) or to [67] and references therein for summetry applied to RMT.”
BackgroundMethodInfluentialQuantum many-body systems out of equilibrium
Eisert, J., Friesdorf, M., & Gogolin, C. (2015) · Nature Physics · doi:10.1038/nphys3215 · Cited by 1,255 · cited 1×
“In this appendix, we consider the case of an additive external noise which can also be important in many situations, in particular in quantum chaos and quantum transport [185], with renewed interest coming fromproblems of quantum ergodicity (‘‘eigenstate thermalization’’) [130,186], entanglement and dissipation (for r…”
Background
Metadata
- Authors
- Bun, J., Bouchaud, J.-P., & Potters, M.
- Venue
- Physics Reports
- Publisher
- Elsevier BV
- Published
- 2017-01
- Volume
- 666
- Pages
- 1-109
- ISSN
- 0370-1573 (ISSN-L)
- OpenAlex
- W2541919214
- Cited by
- 234
- References
- 12
- FWCI
- 12.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.