A COMPARISON OF FLARE FORECASTING METHODS. I. RESULTS FROM THE “ALL-CLEAR” WORKSHOP
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Solar flares produce radiation that can have an almost immediate effect on the near-Earth environment, making it crucial to forecast flares in order to mitigate their negative effects. The number of published approaches to flare forecasting using photospheric magnetic field observations has proliferated, with varying claims about how well each works. Because of the different analysis techniques and data sets used, it is essentially impossible to compare the results from the literature. This problem is exacerbated by the low event rates of large solar flares. The challenges of forecasting rare events have long been recognized in the meteorology community, but have yet to be fully acknowledged by the space weather community. During the interagency workshop on “all clear” forecasts held in Boulder, CO in 2009, the performance of a number of existing algorithms was compared on common data sets, specifically line-of-sight magnetic field and continuum intensity images from the Michelson Doppler Imager, with consistent definitions of what constitutes an event. We demonstrate the importance of making such systematic comparisons, and of using standard verification statistics to determine what constitutes a good prediction scheme. When a comparison was made in this fashion, no one method clearly outperformed all others, which may in part be due to the strong correlations among the parameters used by different methods to characterize an active region. For M-class flares and above, the set of methods tends toward a weakly positive skill score (as measured with several distinct metrics), with no participating method proving substantially better than climatological forecasts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“The results presented here used the open-source SVM implementation called LIBSVM (Chang & Lin 2011).”
MethodThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
The Nature of Statistical Learning Theory
Vapnik, V. N. (2000) · Springer eBooks · doi:10.1007/978-1-4757-3264-1 · Cited by 10,696 · cited 1×
“The larger the distance from the plane to the two different classes of data points in the n-dimensional space, the smaller the classification error (Cortes & Vapnik 1995).”
BackgroundDensity Estimation for Statistics and Data Analysis
(1986) · Springer eBooks · doi:10.1007/978-1-4899-3324-9 · Cited by 8,525
Density Estimation for Statistics and Data Analysis
Dehnad, K. (1987) · Technometrics · doi:10.1080/00401706.1987.10488295 · Cited by 5,302
Probability Theory: The Logic of Science
Jaynes, E. T. (2003) · doi:10.1017/cbo9780511790423 · Cited by 4,045
A leisurely look at the bootstrap, the jackknife, and cross-validation
Efron, B., & Gong, G. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483087 · Cited by 3,229
The Solar Oscillations Investigation - Michelson Doppler Imager
Scherrer, P. H., Bogart, R. S., Bush, R. I., Hoeksema, J. T., Kosovichev, A. G., Schou, J., Rosenberg, W., Springer, L., Tarbell, T. D., Title, A., et al. (1995) · Solar Physics · doi:10.1007/bf00733429 · Cited by 2,393 · cited 1×
“Data from the Solar and Heliospheric Observatory/Michelson Doppler Imager (SOHO/MDI; Scherrer et al. 1995) were prepared and distributed, and it was requested that participants with flare-prediction algorithms use their own methods to make predictions from the data.”
MethodThe Advanced Theory of Statistics.
Watson, G. S., Kendall, M., Stuart, A., & Ord, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288735 · Cited by 921
Proceedings of the fifth annual workshop on Computational learning theory
(1992) · doi:10.1145/130385 · Cited by 560
The Finley Affair: A Signal Event in the History of Forecast Verification
Murphy, A. H. (1996) · Weather and Forecasting · doi:10.1175/1520-0434(1996)011<0003:tfaase>2.0.co;2 · Cited by 219 · cited 2×
“The challenges of forecasting when event rates are low have long been recognized in the meteorology community (e.g., Murphy 1996), but have yet to be fully acknowledged by the space weather community.”
BackgroundThe Evaluation of Yes/No Forecasts for Scientific and Administrative Purposes
Woodcock, F. (1976) · Monthly Weather Review · doi:10.1175/1520-0493(1976)104<1209:teoyff>2.0.co;2 · Cited by 183 · cited 5×
“The use of climatological skill scores (Woodcock 1976; Jolliffe & Stephenson 2003), which account for event climatology and in some cases for underlying sample discrepancies, enables a more informative assessment of forecast performance (Balch 2008; Barnes & Leka 2008; Bloomfield et al. 2012; Crown…”
BackgroundMethodInfluentialBalch, C. C. (2008) · Space Weather · doi:10.1029/2007sw000337 · Cited by 151 · cited 1×
“…of climatological skill scores (Woodcock 1976; Jolliffe & Stephenson 2003), which account for event climatology and in some cases for underlying sample discrepancies, enables a more informative assessment of forecast performance (Balch 2008; Barnes & Leka 2008; Bloomfield et al. 2012; Crown 2012).”
BackgroundSelf-citationA Bayesian Approach to Solar Flare Prediction
Wheatland, M. S. (2004) · The Astrophysical Journal · doi:10.1086/421261 · Cited by 116 · cited 3×
“The event statistics method (Wheatland 2004) predicts flaring probability for different flare sizes using only the flaring history of observed active regions.”
BackgroundMethodInfluentialSelf-citationFalconer, D. A., Barghouty, A. F., Khazanov, I., & Moore, R. L. (2011) · Space Weather · doi:10.1029/2009sw000537 · Cited by 116 · cited 2×
“The modern version has been implemented in a code called MAG-4 (Falconer et al. 2011), running at the Community Coordinated Modeling Center and elsewhere.”
MethodSelf-citationAutomated flare forecasting using a statistical learning technique
Yuan, Y., Shih, F. Y., Jing, J., & Wang, H.-M. (2010) · Research in Astronomy and Astrophysics · doi:10.1088/1674-4527/10/8/008 · Cited by 103
Self-citationA statistical solar flare forecast method
Wheatland, M. S. (2005) · Space Weather · doi:10.1029/2004sw000131 · Cited by 92 · cited 3×
“The whole-Sun method presented in Wheatland (2005) is used in this study, and the method is also applied (for the first time) to events in individual active regions.”
MethodInfluentialSelf-citationAre Solar Active Regions with Major Flares More Fractal, Multifractal, or Turbulent Than Others?
Georgoulis, M. K. (2011) · Solar Physics · doi:10.1007/s11207-010-9705-2 · Cited by 63 · cited 1×
“Georgoulis (2012) show that one multifractal algorithm is resolution-dependent.”
BackgroundSelf-citationSHORT-TERM SOLAR FLARE LEVEL PREDICTION USING A BAYESIAN NETWORK APPROACH
Yu, D., Huang, X., Wang, H., Cui, Y., Hu, Q., & Zhou, R. (2010) · The Astrophysical Journal · doi:10.1088/0004-637x/710/1/869 · Cited by 60 · cited 1×
“…each works (e.g., Abramenko 2005; McAteer et al. 2005a; Jing et al. 2006; Leka & Barnes 2007; Schrijver 2007; Barnes & Leka 2008; Mason & Hoeksema 2010; Yu et al. 2010; Yang et al. 2013; Al- Ghraibah et al. 2015; Boucheron et al. 2015, in addition to references for each method described, below).”
25 Years of Self-organized Criticality: Numerical Detection Methods
McAteer, R. T. J., Aschwanden, M. J., Dimitropoulou, M., Georgoulis, M. K., Pruessner, G., Morales, L., Ireland, J., & Abramenko, V. I. (2015) · Space Science Reviews · doi:10.1007/s11214-015-0158-7 · Cited by 55 · cited 1×
“However, McAteer (2015) and McAteer et al. (2016) show that a multifractal analysis that uses the wavelet transform modulus maxima approach, performed over a series of images, may produce a useful measure of the energy stored in the coronal magnetic field.”
Self-citationMAGNETIC NONPOTENTIALITY IN PHOTOSPHERIC ACTIVE REGIONS AS A PREDICTOR OF SOLAR FLARES
Yang, X. P., Lin, G., Zhang, H., & Mao, X. (2013) · The Astrophysical Journal Letters · doi:10.1088/2041-8205/774/2/l27 · Cited by 28 · cited 1×
“…each works (e.g., Abramenko 2005; McAteer et al. 2005a; Jing et al. 2006; Leka & Barnes 2007; Schrijver 2007; Barnes & Leka 2008; Mason & Hoeksema 2010; Yu et al. 2010; Yang et al. 2013; Al- Ghraibah et al. 2015; Boucheron et al. 2015, in addition to references for each method described, below).”
Cited by
The Origin, Early Evolution and Predictability of Solar Eruptions
Green, L. M., Török, T., Vršnak, B., Manchester, W., & Veronig, A. (2018) · Space Science Reviews · doi:10.1007/s11214-017-0462-5 · Cited by 258
Deep Flare Net (DeFN) Model for Solar Flare Prediction
Nishizuka, N., Sugiura, K., Kubo, Y., Den, M., & Ishii, M. (2018) · The Astrophysical Journal · doi:10.3847/1538-4357/aab9a7 · Cited by 170
“However, there is still discussion on which measure is more suitable for verifying rare-event forecasts (Barnes et al. 2016).”
BackgroundEvent Prediction in the Big Data Era: A Systematic Survey
Zhao, L. (2022) · ACM Computing Surveys · doi:10.1145/3450287 · Cited by 135
Stable reliability diagrams for probabilistic classifiers
Dimitriadis, T., Gneiting, T., & Jordan, A. I. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016191118 · Cited by 84
Metadata
- Authors
- Barnes, G., Leka, K. D., Schrijver, C. J., Colak, T., Qahwaji, R., Ashamari, O. W., Yuan, Y., Zhang, J., McAteer, R. T. J., Bloomfield, D. S., Higgins, P. A., Gallagher, P. T., Falconer, D. A., Georgoulis, M. K., Wheatland, M. S., Balch, C., Dunn, T., & Wagner, E. L.
- Venue
- The Astrophysical Journal
- Publisher
- American Astronomical Society
- Published
- 2016-09-26
- Volume
- 829
- Issue
- 2
- Pages
- 89
- ISSN
- 0004-637X (ISSN-L)
- 1538-4357
- OpenAlex
- W2509145218
- Cited by
- 227
- References
- 22
- FWCI
- 11.07
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.