CoChat

Deep language models for interpretative and predictive materials science

Open access
Hu, Yiwen, Buehler, Markus J. · 2023 · APL Machine Learning
Cited by 78

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

Abstract

Machine learning (ML) has emerged as an indispensable methodology to describe, discover, and predict complex physical phenomena that efficiently help us learn underlying functional rules, especially in cases when conventional modeling approaches cannot be applied. While conventional feedforward neural networks are typically limited to performing tasks related to static patterns in data, recursive models can both work iteratively based on a changing input and discover complex dynamical relationships in the data. Deep language models can model flexible modalities of data and are capable of learning rich dynamical behaviors as they operate on discrete or continuous symbols that define the states of a physical system, yielding great potential toward end-to-end predictions. Similar to how words form a sentence, materials can be considered as a self-assembly of physically interacted building blocks, where the emerging functions of materials are analogous to the meaning of sentences. While discovering the fundamental relationships between building blocks and function emergence can be challenging, language models, such as recurrent neural networks and long-short term memory networks, and, in particular, attention models, such as the transformer architecture, can solve many such complex problems. Application areas of such models include protein folding, molecular property prediction, prediction of material failure of complex nonlinear architected materials, and also generative strategies for materials discovery. We outline challenges and opportunities, especially focusing on extending the deep-rooted kinship of humans with symbolism toward generalizable artificial intelligence (AI) systems using neuro-symbolic AI, and outline how tools such as ChatGPT and DALL·E can drive materials discovery.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1063/5.0134317

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 65

  1. 1931, A Manual of Greek Mathematics
  2. 2011, A History of Mathematics
  3. 1947, Numerical inverting of matrices of high order, Bull. Am. Math. Soc., 53, 1021, 10.1090/s0002-9904-1947-08909-6 · doi:10.1090/s0002-9904-1947-08909-6
  4. 2009, The Birth of Numerical Analysis
  5. 2022, Multiscale modeling at the interface of molecular mechanics and natural language through attention neural networks, Acc. Chem. Res., 55, 3387, 10.1021/acs.accounts.2c00330 · DOI
  6. 2017, Attention is all you need, 5998
  7. 2021, Attention in natural language processing, IEEE Trans. Neural Netw. Learn. Syst., 32, 4291, 10.1109/tnnls.2020.3019893 · DOI · Paper page
  8. 2019, NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning, Proteins: Struct., Funct., Bioinf., 87, 520, 10.1002/prot.25674 · DOI · Paper page
  9. 2022, PRESTO: Rapid protein mechanical strength prediction with an end-to-end deep learning model, Extreme Mech. Lett., 55, 101803, 10.1016/j.eml.2022.101803 · DOI
  10. 2022, End-to-end deep learning model to predict and design secondary structure content of structural proteins, ACS Biomater. Sci. Eng., 8, 1156, 10.1021/acsbiomaterials.1c01343 · DOI
  11. 2022, Rapid prediction of protein natural frequencies using graph neural networks, Digital Discovery, 1, 277, 10.1039/d1dd00007a · DOI
  12. 2022, Ultrafast end-to-end protein structure prediction enables high-throughput exploration of uncharacterized proteins, Proc. Natl. Acad. Sci. U. S. A., 119, e2113348119, 10.1073/pnas.2113348119 · DOI · Paper page
  13. 2022, Rapid mechanical property prediction and de novo design of three-dimensional spider webs through graph and GraphPerceiver neural networks, J. Appl. Phys., 132, 074703, 10.1063/5.0097589 · DOI
  14. 2021, End-to-end deep learning method to predict complete strain and stress tensors for complex hierarchical composite microstructures, J. Mech. Phys. Solids, 154, 104506, 10.1016/j.jmps.2021.104506 · DOI
  15. 2022, End-to-end prediction of multimaterial stress fields and fracture patterns using cycle-consistent adversarial and transformer neural networks, Biomed. Eng. Adv., 4, 100038, 10.1016/j.bea.2022.100038 · DOI
  16. 2022, Generative design, manufacturing, and molecular modeling of 3D architected materials based on natural language input, APL Mater., 10, 041107, 10.1063/5.0082338 · DOI
  17. 2021, Deep learning model to predict complex stress and strain fields in hierarchical composites, Sci. Adv., 7, 1, 10.1126/sciadv.abd7416 · DOI · Paper page
  18. 2022, What’s next for AlphaFold and the AI protein-folding revolution, Nature, 604, 234, 10.1038/d41586-022-00997-5 · DOI · Paper page
  19. 2019, A self-consistent sonification method to translate amino acid sequences into musical compositions and application in protein design using artificial intelligence, ACS Nano, 13, 7471, 10.1021/acsnano.9b02180 · DOI · Paper page
  20. 2020, Using deep learning to predict fracture patterns in crystalline solids, Matter, 3, 197, 10.1016/j.matt.2020.04.019 · DOI · Paper page
  21. 2012, Category theory based solution for the building block replacement problem in materials design, Adv. Eng. Mater., 14, 810, 10.1002/adem.201200109 · DOI
  22. 2021, The trRosetta server for fast and accurate protein structure prediction, Nat. Protoc., 16, 5634, 10.1038/s41596-021-00628-9 · DOI · Paper page
  23. 2019, Harnessing machine learning potentials to understand the functional properties of phase-change materials, MRS Bull., 44, 705, 10.1557/mrs.2019.202 · doi:10.1557/mrs.2019.202
  24. 2022, Machine learning and density functional theory, Nat. Rev. Phys., 4, 357, 10.1038/s42254-022-00470-2 · doi:10.1038/s42254-022-00470-2
  25. Rumelhart, 1986, Learning internal representations by error propagation, 318
  26. 1989, Sigma-Pi learning: On radial basis functions and cortical associative learning
  27. 1994, How to make sigma-pi neural networks perform perfectly on regular training sets, Neural Networks, 7, 1285, 10.1016/0893-6080(94)90009-4 · DOI
  28. 1987, Encoding geometric invariances in higher-order neural networks, 301
  29. 2018, SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, Natural Language Processing (System Demonstrations) (Association for Computational Linguistics, 66
  30. 2021, Learning transferable visual models from natural language supervision, 8748
  31. 1995, Models of natural language understanding, Proc. Natl. Acad. Sci. U. S. A., 92, 9977, 10.1073/pnas.92.22.9977 · DOI
  32. 2020, Exploration and prediction of fluid dynamical systems using auto-encoder technology, Phys. Fluids, 32, 067103, 10.1063/5.0012906 · DOI
  33. 2019, Reconstructing quantum states with generative models, Nat. Mach. Intell., 1, 155, 10.1038/s42256-019-0028-1 · DOI · Paper page
  34. 2018, Learning hard quantum distributions with variational autoencoders, npj Quantum Inf., 4, 28, 10.1038/s41534-018-0077-z · DOI
  35. 2019, Variational autoencoder reconstruction of complex many-body physics, Entropy, 21, 1091, 10.3390/e21111091 · DOI
  36. 2022, Deep learning the functional renormalization group, Phys. Rev. Lett., 129, 136402, 10.1103/physrevlett.129.136402 · DOI
  37. 2019, A review of recurrent neural networks: LSTM cells and network architectures, Neural Comput., 31, 1235, 10.1162/neco_a_01199 · DOI · Paper page
  38. 2016, Ask me anything: Dynamic memory networks for natural language processing, 1378
  39. 2021, An attentive survey of attention models, ACM Trans. Intell. Syst. Technol., 12, 53, 10.1145/3465055 · DOI · Paper page
  40. 2015, Show, attend and tell: Neural image caption generation with visual attention, 2048
  41. 2016, Spider silk: Dynamic mechanics, Nat. Mater., 15, 1054, 10.1038/nmat4721 · DOI
  42. 2016, Nanomechanics of silk: The fundamentals of a strong, tough and versatile material, Nanotechnology, 27, 302001, 10.1088/0957-4484/27/30/302001 · DOI
  43. 2020, Perspectives on three-dimensional printing of self-assembling materials and structures, Curr. Opin. Biomed. Eng., 15, 59, 10.1016/j.cobme.2020.01.003 · DOI
  44. 2021, Offline reinforcement learning as one big sequence modeling problem, 1273
  45. 2022, FieldPerceiver: Domain agnostic transformer model to predict multiscale physical fields and nonlinear material properties through neural ologs, Mater. Today, 57, 9, 10.1016/j.mattod.2022.05.020 · DOI
  46. 2021, Perceiver: General perception with iterative attention, 4651
  47. TurboTransformers: An efficient GPU serving system for transformer models, 389
  48. 2022, MatSciBERT: A materials domain language model for text mining and information extraction, npj Comput. Mater., 8, 102, 10.1038/s41524-022-00784-w · DOI · Paper page
  49. 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
  50. 2019, Named entity recognition and normalization applied to large-scale information extraction from the materials science literature, J. Chem. Inf. Model., 59, 3692, 10.1021/acs.jcim.9b00470 · DOI · Paper page
  51. 2017, Machine-learned and codified synthesis parameters of oxide materials, Sci. Data, 4, 170127, 10.1038/sdata.2017.127 · DOI · Paper page
  52. 2018, Auto-generated materials database of Curie and Néel temperatures via semi-supervised relationship extraction, Sci. Data, 5, 180111, 10.1038/sdata.2018.111 · DOI · Paper page
  53. 2021, Smart constitutive laws: Inelastic homogenization through machine learning, Comput. Methods Appl. Mech. Eng., 373, 113482, 10.1016/j.cma.2020.113482 · DOI · Paper page
  54. 2020, Improved protein structure prediction using predicted interresidue orientations, Proc. Natl. Acad. Sci. U. S. A, 117, 1496, 10.1073/pnas.1914677117 · DOI · Paper page
  55. 2022, Collagen transformer: End-to-End transformer model to predict thermal stability of collagen triple helices using an NLP approach, ACS Biomater. Sci. Eng., 8, 4301, 10.1021/acsbiomaterials.2c00737 · DOI
  56. 2022, End-to-end protein normal mode frequency predictions using language and graph models and application to sonification, ACS Nano, 16, 20656, 10.1021/acsnano.2c07681 · DOI
  57. 2021, Deep learning model to predict fracture mechanisms of graphene, npj 2D Mater. Appl., 5, 48, 10.1038/s41699-021-00228-x · DOI
  58. 2022, A novel approach for studying crack propagation in polycrystalline graphene using machine learning algorithms, Comput. Mater. Sci., 201, 110878, 10.1016/j.commatsci.2021.110878 · DOI
  59. 2022, Modeling atomistic dynamic fracture mechanisms using a progressive transformer diffusion model, J. Appl. Mech., 89, 121009, 10.1115/1.4055730 · DOI
  60. A computational building block approach towards multiscale architected materials analysis and design with application to hierarchical metal metamaterials
  61. 2022, DeepFlames: Neural network-driven self-assembly of flame particles into hierarchical structures, MRS Commun., 12, 257, 10.1557/s43579-022-00171-y · DOI
  62. 2021, The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning, 22574
  63. 2021, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nat. Mach. Intell., 3, 218, 10.1038/s42256-021-00302-5 · DOI · Paper page
  64. 2018, Neural-symbolic VQA: Disentangling reasoning from vision and language understanding, 1031
  65. 2019, Visual concept metaconcept learning, 5001

Cited by

Metadata

Authors
Hu, Y., & Buehler, M. J.
Venue
APL Machine Learning
Published
2023-02-14
Volume
1
Issue
1
ISSN
2770-9019 (ISSN-L)
OpenAlex
W4320713584
Cited by
78
References
37
FWCI
5.49
Type
journal article
Language
en
Open access
diamond
Licence
Funders
  • NIH (U01EB014976, 1R01AR07779)
  • USDA (2021-69012-35978)
  • Army Research Office (W911NF2220213)
  • DOE-SERDP (WP22-3475)
  • ONR (N00014-19-1-2375, N00014-20-1-2189)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.