Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data

In the illustrative two-dimensional training set, a linear classifier that disregards elevation information makes mistakes in the lower-left corner of the feature space.

“If we disregard the elevation information and learn a linear classifier to simply minimize the training errors, we would learn the decision boundary shown using a dotted line in Figure 3a. This classifier would make some mistakes in the lower-left corner of the feature space, where the class confusion is difficult to resolve using a linear separator.”

The limited number of labeled instances available for training or cross-validation can often fail to represent the true nature of relationships in scientific problems.

“For this reason, the limited number of labeled instances available for training or crossvalidation can often fail to represent the true nature of relationships in scientific problems.”

Standard methods for assessing and ensuring generalizability of data science models may break down and lead to misleading conclusions in scientific problems.

“Hence, standard methods for assessing and ensuring generalizability of data science models may break down and lead to misleading conclusions.”

Scientific problems are often under-constrained because they suffer from a paucity of representative training samples while involving a large number of physical variables.

“First, scientific problems are often under-constrained in nature as they suffer from paucity of representative training samples while involving a large number of physical variables.”

Physical variables commonly show complex and non-stationary patterns that dynamically change over time.

“Further, physical variables commonly show complex and non-stationary patterns that dynamically change over time.”

A black-box model that lacks mechanistic understanding cannot be used as a basis for subsequent scientific developments even if it achieves somewhat more accurate performance.

“Hence, even if a black-box model achieves somewhat more accurate performance but lacks the ability to deliver a mechanistic understanding of the underlying processes, it cannot be used as a basis for subsequent scientific developments.”

It is easy to learn spurious relationships that look deceptively good on training and test sets but do not generalize well outside the available labeled data.

“In particular, it is easy to learn spurious relationships that look deceptively good on training and test sets (even after using methods such as cross-validation), but do not generalize well outside the available labeled data.”

Theory-based and data science models suffer from deficiencies when applied in problems where both theory and data are currently lacking.

“Despite their individual strengths, theory-based and data science models suffer from certain deficiencies when applied in problems of great scientific relevance, where both theory and data are currently lacking.”

The paradigm of TGDS has already begun to show promise in scientific problems from diverse disciplines.

“The paradigm of TGDS has already begun to show promise in scientific problems from diverse disciplines.”

• cites [18] [18] J. Kawale, S. Liess, A. Kumar, M. Steinbach, P. Snyder, V. Kumar, A. R. Ganguly, N. F. Samatova, and F. Semazzi, 'A graph-based approach to find teleconnections in climate data,' Statistical Analysis and Data Mining , vol. 6, no. 3, pp. 158-179, 2013. - sourceNot checked yet

  Not checked in the free run.

• cites [20] [20] A. P. Singh, S. Medida, and K. Duraisamy, 'Machine learningaugmented predictive modeling of turbulent separated flows over airfoils,' arXiv preprint arXiv:1608.03990 , 2016. - sourceNot checked yet

  Not checked in the free run.

• cites [22] [22] G. Hautier, C. C. Fischer, A. Jain, T. Mueller, and G. Ceder, 'Finding natures missing ternary oxide compounds using machine learning and density functional theory,' Chemistry of Materials , vol. 22, no. 12, pp. 3762-3767, 2010. - sourceNot checked yet

  Not checked in the free run.

The failure of Google Flu Trends was mainly because the data used for training the model in the first few years was not representative of the trends in subsequent years.

“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”

• cites [15] [15] D. Lazer, R. Kennedy, G. King, and A. Vespignani, 'The Parable of Google Flu: Traps in Big Data Analysis,' Science (New York, N.Y.) , vol. 343, no. 6176, pp. 1203-5, Mar. 2014. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/24626916 - sourceNot checked yet

  Not checked in the free run.

Subsurface flow operates in a regime that is difficult to measure directly using in-situ sensors such as boreholes.

“This is mainly because subsurface flow operates in a regime that is difficult to measure directly using in-situ sensors such as boreholes.”

The paucity of representative samples is one of the prime challenges that differentiates scientific problems from mainstream problems involving Internet-scale data.

“The paucity of representative samples is one of the prime challenges that differentiates scientific problems from mainstream problems involving Internet-scale data such as language translation or object recognition”

Subsurface flow involves complex sub-processes that interact in non-linear ways, which are difficult to encapsulate in current theory-based models.

“In addition, subsurface flow involves a number of complex sub-processes that interact in non-linear ways, which are difficult to encapsulate in current theory-based models [37].”

• cites [37] [37] C. Paniconi and M. Putti, 'Physically based modeling in catchment hydrology at 50: Survey and outlook,' Water Resources Research , vol. 51, no. 9, pp. 7090-7129, 2015. - sourceNot checked yet

  Not checked in the free run.

In settings where processes are not completely understood, theory-based models are often forced to make simplifying assumptions about physical processes, leading to poor performance and difficult comprehension.

“In such settings, theory-based models are often forced to make a number of simplifying assumptions about the physical processes, which not only leads to poor performance but also renders the model difficult to comprehend and analyze.”

An interpretable model grounded by explainable theories stands a better chance at safeguarding against learning spurious patterns from data that lead to non-generalizable performance.

“Further, an interpretable model, that is grounded by explainable theories, stands a better chance at safeguarding against the learning of spurious patterns from the data that lead to non-generalizable performance.”

Our knowledge of subsurface flow using state-of-the-art hydrological models is quite limited.

“However, our knowledge of subsurface flow using state-of-the-art hydrological models is quite limited [36].”

• cites [36] [36] M. Ghasemizade and M. Schirmer, 'Subsurface flow contribution in the hydrological cycle: lessons learned and challenges aheada review,' Environmental earth sciences , vol. 69, no. 2, pp. 707-718, 2013. - sourceNot checked yet

  Not checked in the free run.

Scientific knowledge can be used in the design of model families to restrict the space of models to physically consistent solutions.

“First, scientific knowledge can be used in the design of model families to restrict the space of models to physically consistent solutions, e.g., in the selection of response and loss functions or in the design of model architectures.”

• AssumptionRests on an assumption: Scientific theories used to guide data science models are sufficiently accurate and complete to provide useful constraints for improving model generalization.

Global hydrological models lose physical interpretability due to the large number of model parameters that are difficult to interpret meaningfully with respect to the domain.

“In addition, they also lose physical interpretability due to the large number of model parameters that are difficult to interpret meaningfully with respect to the domain.”

Existing hydrological models make use of a broad range of parameters in several weakly-informed physical equations.

“Due to these challenges, existing hydrological models make use of a broad range of parameters in several weakly-informed physical equations.”

Global hydrological models tend to show poor predictive performance in describing subsurface flow processes.

“Thus, global hydrological models tend to show poor predictive performance in describing subsurface flow processes [38].”

• cites [38] [38] M. F. Bierkens, 'Global hydrology 2015: State, trends, and directions,' Water Resources Research , vol. 51, no. 7, pp. 4923-4947, 2015. - sourceNot checked yet

  Not checked in the free run.

A learning algorithm focused on the space of physically consistent models leads to generalizable and scientifically interpretable models.

“A learning algorithm can then be focused on the space of physically consistent models, leading to generalizable and scientifically interpretable models.”

• AssumptionRests on an assumption: Scientific theories used to guide data science models are sufficiently accurate and complete to provide useful constraints for improving model generalization.

Given a model family, a learning algorithm can be guided to focus on physically consistent solutions.

“Second, given a model family, we can also guide a learning algorithm to focus on physically consistent solutions.”

• AssumptionRests on an assumption: Scientific theories used to guide data science models are sufficiently accurate and complete to provide useful constraints for improving model generalization.

Pruning candidate models that are inconsistent with known scientific principles can significantly reduce the variance of models without likely affecting their bias.

“By pruning candidate models that are inconsistent with known scientific principles (shown as shaded regions in Figure 2), we can significantly reduce the variance of models without likely affecting their bias.”

• cites [39] [39] J. Friedman, T. Hastie, and R. Tibshirani, The elements of statistical learning . Springer series in statistics Springer, Berlin, 2001, vol. 1. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Scientific theories used to guide data science models are sufficiently accurate and complete to provide useful constraints for improving model generalization.

Applying black-box data science models in scientific problems raises issues due to the inadequacy of available data in representing the complex spaces of hypotheses encountered in physical domains.

“If we apply 'black-box' data science models in scientific problems, we would notice a completely different set of issues arising due to the inadequacy of the available data in representing the complex spaces of hypotheses encountered in physical domains.”

Plant trait matrices are highly incomplete because observations about plant traits are sparsely available.

“Since observations about plant traits are sparsely available, such a plant trait matrix would be highly incomplete [48].”

• cites [48] [48] J. Kattge, S. Diaz, S. Lavorel, I. Prentice, P. Leadley, G. B¨ onisch, E. Garnier, M. Westoby, P. B. Reich, I. Wright et al. , 'Try-a global database of plant traits,' Global change biology , vol. 17, no. 9, pp. 2905-2935, 2011. - sourceNot checked yet

  Not checked in the free run.

Neither a data-only nor a theory-only approach can be considered sufficient for knowledge discovery in complex scientific applications.

“Hence, neither a data-only nor a theory-only approach can be considered sufficient for knowledge discovery in complex scientific applications.”

A modified Euler-Lagrange constraint that restricts the space of ground-state density to the density manifold observed in the training set helped in learning accurate and self-consistent ground-state densities.

“To overcome this limitation, a modified Euler-Lagrange constraint was proposed in [25], which restricted the space of n 0 ( r ) to the density manifold observed in the training set. This helped in learning accurate as well as self-consistent ground-state densities using the knowledge contained in the data as well as domain theories.”

• cites [25] [25] L. Li, J. C. Snyder, I. M. Pelaschier, J. Huang, U.-N. Niranjan, P. Duncan, M. Rupp, K.-R. M¨ uller, and K. Burke, 'Understanding machine-learned density functionals,' International Journal of Quantum Chemistry , 2015. -Not checked yet

  Not checked in the free run.

In non-invasive electrophysiological imaging of the heart, a traditional black-box model that only uses data is highly prone to learning spurious patterns given the large parameter space and paucity of labeled examples.

“Given the large space of model parameters and the paucity of labeled examples with ground-truth information, a traditional black-box model that only uses the information contained in the data is highly prone to learning spurious patterns.”

Using computational simulations of theory-based models to initialize ANN models can be especially useful when theory-based models can produce approximate simulations quickly.

“One way to address this challenge is by devising novel pretraining strategies where computational simulations of theory-based models are used to initialize the ANN model. This can be especially useful when theorybased models can produce approximate simulations quickly, e.g., approximate model simulations of turbulent flow (see Example 5).”

Filling missing entries in a plant trait matrix can help understand the characteristics of different plant species and their ability to adapt to varying environmental conditions.

“Filling the missing entries in a plant trait matrix can help us understand the characteristics of different plant species and their ability to adapt to varying environmental conditions.”

Incorporating theory-guided spatial distributions as priors along with externally collected ECG data in a hierarchical Bayesian model provides promising results over traditional data science models.

“Incorporating such theory-guided spatial distributions as priors and using it along with externally collected ECG data in a hierarchical Bayesian model has been shown to provide promising results over traditional data science models [26], [27].”

• cites [26] [26] K. C. Wong, L. Wang, and P. Shi, 'Active model with orthotropic hyperelastic material for cardiac image analysis,' in Functional Imaging and Modeling of the Heart . Springer, 2009, pp. 229-238. -Not checked yet

  Not checked in the free run.

• cites [27] [27] J. Xu, J. L. Sapp, A. R. Dehaghani, F. Gao, M. Horacek, and L. Wang, 'Robust transmural electrophysiological imaging: Integrating sparse and dynamic physiological models into ecg-based inference,' in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015 . Springer, 2015, pp. 519-527. - sourceNot checked yet

  Not checked in the free run.

The derivative of the learned kinetic energy functional can easily leave the space of densities observed in the training set, leading to ill-conditioned solutions especially when the training size is small.

“In particular, the derivative of ˆ T [ n ] can easily leave the space of densities observed in the training set, and thus arrive at ill-conditioned solutions especially when the training size is small.”

Using elevation information to constrain a classifier so that it minimizes training error and produces labels consistent with elevation structure can improve the quality of classification maps.

“To address these challenges, there is an opportunity for improving the quality of classification maps by using the domain knowledge that water bodies have a concave elevation structure. Hence, locations at a lower elevation are filled up first before the water level reaches locations at higher elevations. Thus, if we have access to elevation information (e.g. from bathymetric measurements obtained”

• AssumptionRests on an assumption: Water bodies have a concave elevation structure, so locations at lower elevation are filled up first before the water level reaches locations at higher elevations.

Using the species mean of every attribute as initial values in matrix completion improves the accuracy of predicting plant traits over traditional methods.

“This approach has been shown to provide significant improvements in the accuracy of predicting plant traits over traditional methods [47].”

• cites [47] [47] F. Schrodt, J. Kattge, H. Shan, F. Fazayeli, J. Joswig, A. Banerjee, M. Reichstein, G. B¨ onisch, S. D´ ıaz, J. Dickie et al. , 'Bhpmf-a hierarchical bayesian approach to gap-filling and trait prediction for macroecology and functional biogeography,' Global Ecology and Biogeography , vol. 24, no. 12, pp. 1510-1521, 2015. - sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The species mean provides a robust estimate of the average behavior across all organisms.

The inconsistency of regression models in predicting ground-state density can be traced to their inability to capture functional derivative forms used in the Euler-Lagrange equation.

“This inconsistency can be traced to the inability of regression models in capturing functional derivative forms that are used in Equation 5.”

Liu and colleagues introduced a smoothed minimax concave penalty to Lasso that captured squared differences in regression coefficients between adjacent markers to ensure that the difference in genetic effects between adjacent markers is small.

“In fact, Liu and colleagues [28] introduced a smoothed minimax concave penalty to Lasso that captured squared differences in regression coefficients between adjacent markers to ensure that the difference in genetic effects between adjacent markers is small.”

• cites [28] [28] J. Liu, K. Wang, S. Ma, and J. Huang, 'Accounting for linkage disequilibrium in genome-wide association studies: a penalized regression method,' Statistics and its interface , vol. 6, no. 1, p. 99, 2013. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

A location that has been classified as water for a longer number of time-steps has a higher likelihood of being at a deeper location than a location that has been classified as water less frequently.

“In particular, a location that has been classified as water for a longer number of time-steps has a higher likelihood of being at a deeper location than a location that has been classified as water less frequently.”

In recent work, using sparse group Lasso to model climate variables ensured that if a location is selected, all climate variables observed at that location are used as relevant features.

“By treating the set of variables observed at every location as a group, the use of group Lasso ensured that if a location is selected, all of the climate variables observed at that location will be used as relevant features.”

• cites [66] [66] S. Chatterjee, K. Steinhaeuser, A. Banerjee, S. Chatterjee, and A. R. Ganguly, 'Sparse group lasso: Consistency and climate applications.' in SDM . SIAM, 2012, pp. 47-58. - sourceNot checked yet

  Not checked in the free run.

This line of research has resulted in the discovery of a hundred new ternary oxide compounds that were previously unknown using traditional approaches.

“This line of research has resulted in the discovery of a hundred new ternary oxide compounds that were previously unknown using traditional approaches [22], highlighting the effectiveness of TGDS in advancing scientific knowledge.”

• cites [22] [22] G. Hautier, C. C. Fischer, A. Jain, T. Mueller, and G. Ceder, 'Finding natures missing ternary oxide compounds using machine learning and density functional theory,' Chemistry of Materials , vol. 22, no. 12, pp. 3762-3767, 2010. - sourceNot checked yet

  Not checked in the free run.

Use of elevation-based constraints can help in learning a generalizable classification model even with poorly labeled training data.

“The use of such constraints can help in learning a generalizable classification model even with poorly labeled training data.”

While predicting elastic modulus using bond energy and melting point, Lasso may favor melting point over bond energy even though a direct causal link exists between bond energy and the modulus.

“For example, while predicting the elastic modulus using bond energy and melting point, Lasso may favor melting point over bond energy even though a direct causal link exists between bond energy and the modulus [31].”

• cites [31] [31] N. Wagner and J. M. Rondinelli, 'Theory-guided machine learning in materials science,' Frontiers in Materials , vol. 3, p. 28, 2016. -Not checked yet

  Not checked in the free run.

Domain knowledge that relevant genetic markers are typically located in close proximity on the genome due to linkage disequilibrium can be incorporated as a regularizer to ensure discovered genetic markers are located in close proximity.

“This domain knowledge can be incorporated as a regularizer to ensure that the discovered genetic markers are typically located in close proximity on the genome.”

Group Lasso helps in selecting physically meaningful groups of attributes in data science models.

“Group Lasso helps in selecting physically meaningful groups of attributes in the data science models, and various extensions of group Lasso have been explored for handling different types of domain characteristics, e.g., overlapping group Lasso [63], tree-guided group Lasso [64], and sparse group Lasso [65].”

• cites [62] [62] M. Yuan and Y. Lin, 'Model selection and estimation in regression with grouped variables,' Journal of the Royal Statistical Society: Series B (Statistical Methodology) , vol. 68, no. 1, pp. 49-67, 2006. - sourceNot checked yet

  Not checked in the free run.

Creating hybrid combinations of theory-based and data science models, where some aspects are handled by theory-based components and the remaining ones are modeled using data science components, is one way to combine the strengths of scientific knowledge and data science.

“One way to combine the strengths of scientific knowledge and data science is by creating hybrid combinations of theory-based and data science models, where some aspects of the problem are handled by theory-based components while the remaining ones are modeled using data science components.”

This implicit elevation ordering, if extracted effectively, can help in improving classification maps by postprocessing outputs to be consistent with elevation ordering.

“This implicit elevation ordering, if extracted effectively, can help in improving the classification maps by postprocessing the outputs to be consistent with elevation ordering.”

Outputs of theory-based models may be used as training samples in data science components, creating a two-way synergy between them.

“Further, the outputs of theory-based models may also be used as training samples in data science components [72], thus creating a two-way synergy between them.”

• cites [72] [72] P. Sadowski, D. Fooshee, N. Subrahmanya, and P. Baldi, 'Synergies between quantum mechanics and machine learning in reaction prediction,' Journal of Chemical Information and Modeling , vol. 56, no. 11, pp. 2125-2128, 2016. sourceNot checked yet

  Not checked in the free run.

Both Wang et al.'s and Singh et al.'s works illustrate the potential of coupling data science outputs with theory-based models to reduce model discrepancies in complex scientific applications.

“Both these works illustrate the potential of coupling data science outputs with theory-based models to reduce model discrepancies in complex scientific applications.”

• AssumptionRests on an assumption: Current RANS models are still insufficient for modeling complex flows with separation, curvature, or swirling.

This approach was successfully used to build global maps of surface water dynamics.

“This approach was successfully used in [29], [30] to build global maps of surface water dynamics.”

• cites [29] [29] A. Khandelwal, V. Mithal, and V. Kumar, 'Post classification label refinement using implicit ordering constraint among data instances,' in Data Mining (ICDM), 2015 IEEE International Conference on . IEEE, 2015, pp. 799-804. - sourceNot checked yet

  Not checked in the free run.

• cites [30] [30] A. Khandelwal, A. Karpatne, M. Marlier, J. Kim, D. Lettenmaier, and V. Kumar, 'An approach for global monitoring of surface water extent variations using modis data,' in Remote Sensing of Environment (in review) , 2017. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Theory-based model outputs can be used to supervise the training of data science models by providing physically consistent estimates of the target variable for every training instance.

“Theory-based model outputs can also be used to supervise the training of data science models, by providing physically consistent estimates of the target variable for every training instance.”

Data can be assimilated in theory-based models for improved selection of model states in numerical models.

“Data can be assimilated in theory-based models for improved selection of model states in numerical models.”

• cites [74] [74] G. Evensen, Data assimilation: the ensemble Kalman filter . Springer Science & Business Media, 2009. sourceNot checked yet

  Not checked in the free run.

Using data science methods to predict intermediate quantities in theory-based models that are currently being missed or inaccurately estimated can create a hybrid model that shows better predictive performance and amends deficiencies in existing theory-based models.

“An alternate way of creating a hybrid TGDS model is to use data science methods to predict intermediate quantities in theory-based models that are currently being missed or inaccurately estimated. By feeding data science outputs into theory-based models, such a hybrid model can not only show better predictive performance but also amend the deficiencies in existing theory-based models.”

Variants of multi-armed bandit techniques have been explored for settings where parameters take continuous values instead of discrete steps.

“Variants of these techniques have also been explored for settings where the parameters take continuous values instead of discrete steps [79], [80].”

• cites [79] [79] R. Agrawal, 'The continuum-armed bandit problem,' SIAM journal on control and optimization , vol. 33, no. 6, pp. 1926-1951, 1995.Not checked yet

  Not checked in the free run.

• cites [80] [80] R. D. Kleinberg, 'Nearly tight bounds for the continuum-armed bandit problem,' in Advances in Neural Information Processing Systems , 2004, pp. 697-704.Not checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: Theory-based models often involve a large number of parameters in their equations that need to be calibrated in order to provide an accurate representation of the physical system.

A naïve approach for model calibration-trying every combination of parameter values and choosing the combination that produces maximum likelihood-is practically infeasible when the number of parameters is large and every parameter takes many possible values.

“However, this approach is practically infeasible when the number of parameters are large and every parameter takes many possible values.”

• AssumptionRests on an assumption: Theory-based models often involve a large number of parameters in their equations that need to be calibrated in order to provide an accurate representation of the physical system.