Deep language models for interpretative and predictive materials science
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Machine learning (ML) has emerged as an indispensable methodology to describe, discover, and predict complex physical phenomena that efficiently help us learn underlying functional rules, especially in cases when conventional modeling approaches cannot be applied. While conventional feedforward neural networks are typically limited to performing tasks related to static patterns in data, recursive models can both work iteratively based on a changing input and discover complex dynamical relationships in the data. Deep language models can model flexible modalities of data and are capable of learning rich dynamical behaviors as they operate on discrete or continuous symbols that define the states of a physical system, yielding great potential toward end-to-end predictions. Similar to how words form a sentence, materials can be considered as a self-assembly of physically interacted building blocks, where the emerging functions of materials are analogous to the meaning of sentences. While discovering the fundamental relationships between building blocks and function emergence can be challenging, language models, such as recurrent neural networks and long-short term memory networks, and, in particular, attention models, such as the transformer architecture, can solve many such complex problems. Application areas of such models include protein folding, molecular property prediction, prediction of material failure of complex nonlinear architected materials, and also generative strategies for materials discovery. We outline challenges and opportunities, especially focusing on extending the deep-rooted kinship of humans with symbolism toward generalizable artificial intelligence (AI) systems using neuro-symbolic AI, and outline how tools such as ChatGPT and DALL·E can drive materials discovery.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Accurate prediction of protein structures and interactions using a 3-track network
Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. (2021) · doi:10.1101/2021.06.14.448402 · Cited by 5,549
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Deep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 3,485
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329
E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials
Batzner, S., Musaelian, A., Sun, L., Geiger, M., Mailoa, J. P., Kornbluth, M., Molinari, N., Smidt, T. E., & Kozinsky, B. (2022) · Nature Communications · doi:10.1038/s41467-022-29939-5 · Cited by 2,285
Unke, O. T., Chmiela, S., Sauceda, H. E., Gastegger, M., Poltavsky, I., Schütt, K. T., Tkatchenko, A., & Müller, K.-R. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.0c01111 · Cited by 1,647
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
Unsupervised word embeddings capture latent knowledge from materials science literature
Tshitoyan, V., Dagdelen, J., Weston, L., Dunn, A., Rong, Z., Kononova, O., Persson, K. A., Ceder, G., & Jain, A. (2019) · Nature · doi:10.1038/s41586-019-1335-8 · Cited by 1,123
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Artificial intelligence and machine learning in design of mechanical materials
Guo, K., Yang, Z., Yu, C.-H., & Buehler, M. J. (2021) · Materials Horizons · doi:10.1039/d0mh01451f · Cited by 735
Self-citationGalactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
Swain, M. C., & Cole, J. M. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00207 · Cited by 598
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019) · arXiv · doi:10.48550/arxiv.1904.12584 · Cited by 444
Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02272-z · Cited by 439
End-to-End Differentiable Learning of Protein Structure
AlQuraishi, M. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.006 · Cited by 371
Data-driven materials research enabled by natural language processing and information extraction
Olivetti, E. A., Cole, J. M., Kim, E., Kononova, O., Ceder, G., Han, T. Y.-J., & Hiszpanski, A. M. (2020) · Applied Physics Reviews · doi:10.1063/5.0021106 · Cited by 302
Protein structure generation via folding diffusion
Wu, K. E., Yang, K. K., Berg, R. V. D., Zou, J. Y., Lu, A. X., & Amini, A. P. (2022) · arXiv · doi:10.48550/arxiv.2209.15611 · Cited by 193
Deep learning techniques have significantly impacted protein structure prediction and protein design
Pearce, R., & Zhang, Y. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.01.007 · Cited by 127
Design in the DARK: Learning Deep Generative Models for De Novo Protein Design
Moffat, L., Kandathil, S. M., & Jones, D. T. (2022) · doi:10.1101/2022.01.27.478087 · Cited by 25
Meta Learning Backpropagation And Improving It
Kirsch, L., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.14905 · Cited by 19
Research works printed but not linked: 65
- 1931, A Manual of Greek Mathematics
- 2011, A History of Mathematics
- 1947, Numerical inverting of matrices of high order, Bull. Am. Math. Soc., 53, 1021, 10.1090/s0002-9904-1947-08909-6 · doi:10.1090/s0002-9904-1947-08909-6
- 2009, The Birth of Numerical Analysis
- 2022, Multiscale modeling at the interface of molecular mechanics and natural language through attention neural networks, Acc. Chem. Res., 55, 3387, 10.1021/acs.accounts.2c00330 · DOI
- 2017, Attention is all you need, 5998
- 2021, Attention in natural language processing, IEEE Trans. Neural Netw. Learn. Syst., 32, 4291, 10.1109/tnnls.2020.3019893 · DOI · Paper page
- 2019, NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning, Proteins: Struct., Funct., Bioinf., 87, 520, 10.1002/prot.25674 · DOI · Paper page
- 2022, PRESTO: Rapid protein mechanical strength prediction with an end-to-end deep learning model, Extreme Mech. Lett., 55, 101803, 10.1016/j.eml.2022.101803 · DOI
- 2022, End-to-end deep learning model to predict and design secondary structure content of structural proteins, ACS Biomater. Sci. Eng., 8, 1156, 10.1021/acsbiomaterials.1c01343 · DOI
- 2022, Rapid prediction of protein natural frequencies using graph neural networks, Digital Discovery, 1, 277, 10.1039/d1dd00007a · DOI
- 2022, Ultrafast end-to-end protein structure prediction enables high-throughput exploration of uncharacterized proteins, Proc. Natl. Acad. Sci. U. S. A., 119, e2113348119, 10.1073/pnas.2113348119 · DOI · Paper page
- 2022, Rapid mechanical property prediction and de novo design of three-dimensional spider webs through graph and GraphPerceiver neural networks, J. Appl. Phys., 132, 074703, 10.1063/5.0097589 · DOI
- 2021, End-to-end deep learning method to predict complete strain and stress tensors for complex hierarchical composite microstructures, J. Mech. Phys. Solids, 154, 104506, 10.1016/j.jmps.2021.104506 · DOI
- 2022, End-to-end prediction of multimaterial stress fields and fracture patterns using cycle-consistent adversarial and transformer neural networks, Biomed. Eng. Adv., 4, 100038, 10.1016/j.bea.2022.100038 · DOI
- 2022, Generative design, manufacturing, and molecular modeling of 3D architected materials based on natural language input, APL Mater., 10, 041107, 10.1063/5.0082338 · DOI
- 2021, Deep learning model to predict complex stress and strain fields in hierarchical composites, Sci. Adv., 7, 1, 10.1126/sciadv.abd7416 · DOI · Paper page
- 2022, What’s next for AlphaFold and the AI protein-folding revolution, Nature, 604, 234, 10.1038/d41586-022-00997-5 · DOI · Paper page
- 2019, A self-consistent sonification method to translate amino acid sequences into musical compositions and application in protein design using artificial intelligence, ACS Nano, 13, 7471, 10.1021/acsnano.9b02180 · DOI · Paper page
- 2020, Using deep learning to predict fracture patterns in crystalline solids, Matter, 3, 197, 10.1016/j.matt.2020.04.019 · DOI · Paper page
- 2012, Category theory based solution for the building block replacement problem in materials design, Adv. Eng. Mater., 14, 810, 10.1002/adem.201200109 · DOI
- 2021, The trRosetta server for fast and accurate protein structure prediction, Nat. Protoc., 16, 5634, 10.1038/s41596-021-00628-9 · DOI · Paper page
- 2019, Harnessing machine learning potentials to understand the functional properties of phase-change materials, MRS Bull., 44, 705, 10.1557/mrs.2019.202 · doi:10.1557/mrs.2019.202
- 2022, Machine learning and density functional theory, Nat. Rev. Phys., 4, 357, 10.1038/s42254-022-00470-2 · doi:10.1038/s42254-022-00470-2
- Rumelhart, 1986, Learning internal representations by error propagation, 318
- 1989, Sigma-Pi learning: On radial basis functions and cortical associative learning
- 1994, How to make sigma-pi neural networks perform perfectly on regular training sets, Neural Networks, 7, 1285, 10.1016/0893-6080(94)90009-4 · DOI
- 1987, Encoding geometric invariances in higher-order neural networks, 301
- 2018, SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, Natural Language Processing (System Demonstrations) (Association for Computational Linguistics, 66
- 2021, Learning transferable visual models from natural language supervision, 8748
- 1995, Models of natural language understanding, Proc. Natl. Acad. Sci. U. S. A., 92, 9977, 10.1073/pnas.92.22.9977 · DOI
- 2020, Exploration and prediction of fluid dynamical systems using auto-encoder technology, Phys. Fluids, 32, 067103, 10.1063/5.0012906 · DOI
- 2019, Reconstructing quantum states with generative models, Nat. Mach. Intell., 1, 155, 10.1038/s42256-019-0028-1 · DOI · Paper page
- 2018, Learning hard quantum distributions with variational autoencoders, npj Quantum Inf., 4, 28, 10.1038/s41534-018-0077-z · DOI
- 2019, Variational autoencoder reconstruction of complex many-body physics, Entropy, 21, 1091, 10.3390/e21111091 · DOI
- 2022, Deep learning the functional renormalization group, Phys. Rev. Lett., 129, 136402, 10.1103/physrevlett.129.136402 · DOI
- 2019, A review of recurrent neural networks: LSTM cells and network architectures, Neural Comput., 31, 1235, 10.1162/neco_a_01199 · DOI · Paper page
- 2016, Ask me anything: Dynamic memory networks for natural language processing, 1378
- 2021, An attentive survey of attention models, ACM Trans. Intell. Syst. Technol., 12, 53, 10.1145/3465055 · DOI · Paper page
- 2015, Show, attend and tell: Neural image caption generation with visual attention, 2048
- 2016, Spider silk: Dynamic mechanics, Nat. Mater., 15, 1054, 10.1038/nmat4721 · DOI
- 2016, Nanomechanics of silk: The fundamentals of a strong, tough and versatile material, Nanotechnology, 27, 302001, 10.1088/0957-4484/27/30/302001 · DOI
- 2020, Perspectives on three-dimensional printing of self-assembling materials and structures, Curr. Opin. Biomed. Eng., 15, 59, 10.1016/j.cobme.2020.01.003 · DOI
- 2021, Offline reinforcement learning as one big sequence modeling problem, 1273
- 2022, FieldPerceiver: Domain agnostic transformer model to predict multiscale physical fields and nonlinear material properties through neural ologs, Mater. Today, 57, 9, 10.1016/j.mattod.2022.05.020 · DOI
- 2021, Perceiver: General perception with iterative attention, 4651
- TurboTransformers: An efficient GPU serving system for transformer models, 389
- 2022, MatSciBERT: A materials domain language model for text mining and information extraction, npj Comput. Mater., 8, 102, 10.1038/s41524-022-00784-w · DOI · Paper page
- 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
- 2019, Named entity recognition and normalization applied to large-scale information extraction from the materials science literature, J. Chem. Inf. Model., 59, 3692, 10.1021/acs.jcim.9b00470 · DOI · Paper page
- 2017, Machine-learned and codified synthesis parameters of oxide materials, Sci. Data, 4, 170127, 10.1038/sdata.2017.127 · DOI · Paper page
- 2018, Auto-generated materials database of Curie and Néel temperatures via semi-supervised relationship extraction, Sci. Data, 5, 180111, 10.1038/sdata.2018.111 · DOI · Paper page
- 2021, Smart constitutive laws: Inelastic homogenization through machine learning, Comput. Methods Appl. Mech. Eng., 373, 113482, 10.1016/j.cma.2020.113482 · DOI · Paper page
- 2020, Improved protein structure prediction using predicted interresidue orientations, Proc. Natl. Acad. Sci. U. S. A, 117, 1496, 10.1073/pnas.1914677117 · DOI · Paper page
- 2022, Collagen transformer: End-to-End transformer model to predict thermal stability of collagen triple helices using an NLP approach, ACS Biomater. Sci. Eng., 8, 4301, 10.1021/acsbiomaterials.2c00737 · DOI
- 2022, End-to-end protein normal mode frequency predictions using language and graph models and application to sonification, ACS Nano, 16, 20656, 10.1021/acsnano.2c07681 · DOI
- 2021, Deep learning model to predict fracture mechanisms of graphene, npj 2D Mater. Appl., 5, 48, 10.1038/s41699-021-00228-x · DOI
- 2022, A novel approach for studying crack propagation in polycrystalline graphene using machine learning algorithms, Comput. Mater. Sci., 201, 110878, 10.1016/j.commatsci.2021.110878 · DOI
- 2022, Modeling atomistic dynamic fracture mechanisms using a progressive transformer diffusion model, J. Appl. Mech., 89, 121009, 10.1115/1.4055730 · DOI
- A computational building block approach towards multiscale architected materials analysis and design with application to hierarchical metal metamaterials
- 2022, DeepFlames: Neural network-driven self-assembly of flame particles into hierarchical structures, MRS Commun., 12, 257, 10.1557/s43579-022-00171-y · DOI
- 2021, The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning, 22574
- 2021, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nat. Mach. Intell., 3, 218, 10.1038/s42256-021-00302-5 · DOI · Paper page
- 2018, Neural-symbolic VQA: Disentangling reasoning from vision and language understanding, 1031
- 2019, Visual concept metaconcept learning, 5001
Cited by
Lu, W., Luu, R. K., & Buehler, M. J. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01564-y · Cited by 126
Self-citationBuehler, M. J. (2024) · ACS Engineering Au · doi:10.1021/acsengineeringau.3c00058 · Cited by 82
Self-citationAutomating alloy design and discovery with physics-aware multimodal multiagent AI
Ghafarollahi, A., & Buehler, M. J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414074122 · Cited by 72
Self-citation
Metadata
- Authors
- Hu, Y., & Buehler, M. J.
- Venue
- APL Machine Learning
- Publisher
- AIP Publishing
- Published
- 2023-02-14
- Volume
- 1
- Issue
- 1
- ISSN
- 2770-9019 (ISSN-L)
- OpenAlex
- W4320713584
- Cited by
- 78
- References
- 37
- FWCI
- 5.49
- Type
- journal article
- Language
- en
- Open access
- diamond
- Licence
- creativecommons.org/licenses/by/4.0/ (vor)
- creativecommons.org/licenses/by/4.0/ (tdm)
- Funders
- NIH (U01EB014976, 1R01AR07779)
- USDA (2021-69012-35978)
- Army Research Office (W911NF2220213)
- DOE-SERDP (WP22-3475)
- ONR (N00014-19-1-2375, N00014-20-1-2189)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.