CoChat
Sign Up
Cited by 60Advanced Neural Network ApplicationsTopic ModelingIoT and Edge/Fog Computing

Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions

Open access
Lamaakal, Ismail, Maleh, Yassine, El Makkaoui, Khalid, Ouahbi, Ibrahim, Pławiak, Paweł, Alfarraj, Osama, Almousa, May, Abd El-Latif, Ahmed A.·2025·Sensors
DOI 10.3390/s25051318PMC PMC11902656
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs), like GPT and BERT, have significantly advanced Natural Language Processing (NLP), enabling high performance on complex tasks. However, their size and computational needs make LLMs unsuitable for deployment on resource-constrained devices, where efficiency, speed, and low power consumption are critical. Tiny Language Models (TLMs), also known as BabyLMs, offer compact alternatives by using advanced compression and optimization techniques to function effectively on devices such as smartphones, Internet of Things (IoT) systems, and embedded platforms. This paper provides a comprehensive survey of TLM architectures and methodologies, including key techniques such as knowledge distillation, quantization, and pruning. Additionally, it explores potential and emerging applications of TLMs in automation and control, covering areas such as edge computing, IoT, industrial automation, and healthcare. The survey discusses challenges unique to TLMs, such as trade-offs between model size and accuracy, limited generalization, and ethical considerations in deployment. Future research directions are also proposed, focusing on hybrid compression techniques, application-specific adaptations, and context-aware TLMs optimized for hardware-specific constraints. This paper aims to serve as a foundational resource for advancing TLMs capabilities across diverse real-world applications.

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews Tiny Language Models (TLMs) as compact, resource-efficient alternatives to LLMs for deployment on smartphones, IoT, embedded systems, and automation/control settings. It covers compression techniques, architectures, datasets, applications, challenges, and future directions, but presents a qualitative overview rather than new experiments.

Key findings

  • TLMs typically contain tens to hundreds of millions of parameters, compared with billions in LLMs such as GPT-3’s 175 billion parameters, making them suitable for resource-constrained devices.
  • Core size-reduction techniques include knowledge distillation, quantization, pruning/sparsification, and efficient architectures; Table 1 defines “High” model-size reduction as >50%, “High” latency improvement as >2×, and “Minimal” accuracy loss as <1%.
  • Reported examples include MobileBERT at ~25.3 million parameters, 4.3× smaller and 5.5× faster than BERT_BASE, with a GLUE score of 77.7 and 62 ms latency on a Pixel 4; TinyBERT at ~14.5 million parameters; DistilBERT at ~66 million parameters; TinyLlama at 1.1 B parameters; and SmolLM at 135 M, 360 M, and 1.7 B parameters.
  • The survey identifies applications in edge computing/IoT, industrial natural-language interfaces, diagnostics/predictive maintenance, and healthcare, and proposes future work on hybrid compression, application-specific TLMs, and context-aware/hardware-specific models.

Methods

  • Design: a comprehensive survey/overview of TLM architectures, optimization methods, training datasets, applications, challenges, and future research directions.
  • Data/sample: reviews existing literature and models, identifies 12 open-sourced pre-training datasets used for TLMs, and compares popular TLM architectures in Table 3.
  • Analysis: narrative and qualitative synthesis; no primary experiments, benchmark runs, or statistical meta-analysis are reported.

Limitations

  • The paper states challenges including trade-offs between model size and accuracy, limited generalization, data efficiency and training constraints, ethical/security concerns, and reliability in real-time applications.
  • Evident from the text: the survey does not conduct primary experiments or quantitative meta-analysis, and its comparisons rely on cited literature and qualitative tables rather than independent benchmarking.
  • Evident from the text: despite claiming to be the first comprehensive TLM survey, no formal systematic-review protocol or inclusion/exclusion criteria are described.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s25051318

Cites

99 of 99 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 64 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The paper's analysis identifies 12 open-sourced pre-training datasets commonly used in training TLMs.

    “Our analysis identifies 12 datasets employed in these efforts:”
    Quote in the paperNot a substantial claim
  • Interpretationreasoning

    The dependency of LLMs on powerful infrastructure makes them primarily suited for centralized, cloud-based applications.

    “This dependency on powerful infrastructure makes LLMs primarily suited for centralized, cloud-based applications, where latency and resource costs can be managed.”
    Quote in the paperSubstantial claim
  • Premisecitation

    In domains like industrial automation, healthcare, and consumer electronics, TLMs allow NLP functionalities to be embedded directly within devices, enabling intelligent interactions without constant reliance on the cloud.

    “In domains like industrial automation [42], healthcare [43], and consumer electronics [44], TLMs allow NLP functionalities to be embedded directly within devices, enabling intelligent interactions without constant reliance on the cloud.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [42] Bangemann, T., Karnouskos, S., Camp, R., Carlsson, O., Riedl, M., McLeod, S., Harrison, R., Colombo, A.W., and Stluka, P. (2014). State of the Art in Industrial Automation. Industrial Cloud-Based Cyber-Physical Systems: The IMC-AESOP Approach, Springer. source

      Not checked in the free run.

    • cites [43] Wang, X., Dang, T., Kostakos, V., and Jia, H. (2024). Efficient and Personalized Mobile Health Event Prediction via Small Language Models. arXiv. source

      Not checked in the free run.

    • cites [44] Wu, 2022, State-of-the-Art and Research Opportunities for Next-Generation Consumer Electronics, IEEE Trans. Consum. Electron. source

      Not checked in the free run.

  • Premisecitation

    TLMs are significantly smaller than LLMs, typically containing tens to hundreds of millions of parameters compared to billions in LLMs like GPT-3.

    “TLMs are significantly smaller than LLMs, typically containing tens to hundreds of millions of parameters compared to billions in LLMs like GPT-3 [4].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4] Kalyan, 2023, A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4, Nat. Lang. Process. J. source

      Not checked in the free run.

  • Premisecitation

    TLMs are typically created by employing techniques such as knowledge distillation, pruning, quantization, and specialized architectures designed to minimize model size and memory footprint.

    “TLMs are typically created by employing techniques such as knowledge distillation, pruning, quantization [17,18,19,20,21,22,23], and specialized architectures [24] designed to minimize model size and memory footprint.”
    Quote in the paperNot a substantial claim0 of 8 sources checked
    • cites [17] Han, S., Mao, H., and Dally, W.J. (2015). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv.

      Not checked in the free run.

    • cites [18] Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. (2023). A Survey on Model Compression for Large Language Models. arXiv. source

      Not checked in the free run.

    • cites [19] Xu, C., and McAuley, J. (2023, January 7–14). A Survey on Model Compression and Acceleration for Pretrained Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.

      Not checked in the free run.

    • cites [20] Lamaakal, 2024, A TinyDL Model for Gesture-Based Air Handwriting Arabic Numbers and Simple Arabic Letters Recognition, IEEE Access source

      Not checked in the free run.

    • cites [21] Lamaakal, I., Maleh, Y., Ouahbi, I., El Makkaoui, K., and Abd El-Latif, A.A. (2024, January 30–31). A Deep Learning-Powered TinyML Model for Gesture-Based Air Handwriting Simple Arabic Letters Recognition. Proceedings of the International Conference on Digital Technologies and Applications, Ningbo, China. source

      Not checked in the free run.

    • cites [22] Lamaakal, I., El Mourabit, N., El Makkaoui, K., Ouahbi, I., and Maleh, Y. (2024, January 23–24). Efficient Gesture-Based Recognition of Tifinagh Characters in Air Handwriting with a TinyDL Model. Proceedings of the 2024 Sixth International Conference on Intelligent Computing in Data Sciences (ICDS), Marrakech, Morocco. source

      Not checked in the free run.

    • cites [23] Lamaakal, 2024, A TinyML Model for Gesture-Based Air Handwriting Arabic Numbers Recognition, Procedia Comput. Sci. source

      Not checked in the free run.

    • cites [24] Tang, Y., Liu, F., Ni, Y., Tian, Y., Bai, Z., Hu, Y.Q., Liu, S., Jui, S., Han, K., and Wang, Y. (2024). Rethinking Optimization and Architecture for Tiny Language Models. arXiv.

      Not checked in the free run.

  • May combine two claims. 39
  • Classified as probably not a substantial claim. 22

  • May combine two claims.

    Premisecitation

    LLMs require substantial memory, processing power, and storage, often relying on high-end GPUs or specialized hardware like TPUs for training and inference.

    “LLMs require substantial memory, processing power, and storage space [14], often relying on high-end GPUs or specialized hardware like TPUs (Tensor Processing Units) for both training and inference [15].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14] Patil, R., and Gudivada, V. (2024). A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs). Appl. Sci., 14. source

      Not checked in the free run.

    • cites [15] Bai, G., Chai, Z., Ling, C., Wang, S., Lu, J., Zhang, N., Shi, T., Yu, Z., Zhu, M., and Zhang, Y. (2024). Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models. arXiv.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    TLMs can operate on standard CPUs or low-power GPUs, making them ideal for edge devices and resource-constrained environments.

    “They can operate on standard CPUs or low-power GPUs, making them ideal for edge devices and resource-constrained environments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    These methods enable TLMs to operate with a fraction of the parameters found in LLMs, making them more suitable for deployment on devices with limited resources, such as smartphones, embedded systems, and industrial IoT sensors.

    “These methods enable TLMs to operate with a fraction of the parameters found in LLMs, making them more suitable for deployment on devices with limited resources, such as smartphones, embedded systems, and industrial IoT sensors [25].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [25] Samie, F., Bauer, L., and Henkel, J. (2016, January 1–7). IoT Technologies for Embedded Computing: A Survey. Proceedings of the Eleventh IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis, Pittsburgh, PA, USA. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TLMs are versatile and can be adapted for a wide range of tasks with minimal fine-tuning.

    “TLMs are versatile and can be adapted for a wide range of tasks with minimal fine-tuning [71].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [71] Mitchell, E., Rafailov, R., Sharma, A., Finn, C., and Manning, C.D. (2023). An Emulator for Fine-Tuning Large Language Models Using Small Language Models. arXiv.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    GPT-3, with 175 billion parameters, necessitates vast computing resources and is impractical to deploy directly on mobile devices, edge systems, or IoT platforms.

    “For example, the popular GPT-3 model [4], with 175 billion parameters, necessitates vast computing resources and is impractical to deploy directly on mobile devices, edge systems, or IoT platforms [16].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [4] Kalyan, 2023, A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4, Nat. Lang. Process. J. source

      Not checked in the free run.

    • cites [16] Bhardwaj, S., Singh, P., and Pandit, M.K. (2024, January 14–16). A Survey on the Integration and Optimization of Large Language Models in Edge Computing Environments. Proceedings of the 2024 16th International Conference on Computer and Automation Engineering (ICCAE), Melbourne, Australia. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TLMs require less memory and processing power, which translates to lower energy consumption.

    “TLMs are designed for efficiency, requiring less memory and processing power, which translates to lower energy consumption [70].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [70] Scherer, M., Macan, L., Jung, V.J., Wiese, P., Bompani, L., Burrello, A., Conti, F., and Benini, L. (2024). Deeploy: Enabling Energy-Efficient Deployment of Small Language Models on Heterogeneous Microcontrollers. arXiv. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    On-device deployment of TLMs ensures sensitive information is processed locally, reducing risks associated with data transmission and improving data privacy.

    “Since they can be deployed directly on devices, there is minimal need to send data to the cloud for processing. This on-device capability ensures that sensitive information is processed locally, reducing potential risks associated with data transmission and improving data privacy [74].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [74] Das, 2024, Security and Privacy Challenges of Large Language Models: A Survey, ACM Comput. Surv.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TLMs enable on-device natural language understanding, allowing smart home devices to interpret voice commands, automate routines, and interact with users efficiently.

    “TLMs enable on-device natural language understanding, allowing smart home devices to interpret voice commands, automate routines, and interact with users efficiently [142].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [142] Mardikyan, 2017, Understanding the Adoption of Voice Activated Personal Assistants, Int. J. E-Serv. Mob. Appl. (IJESMA) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    TLMs enable personalized health insights, generating natural language summaries of medical data and assisting in early disease detection through on-device AI processing.

    “Additionally, TLMs enable personalized health insights, generating natural language summaries of medical data and assisting in early disease detection through on-device AI processing.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    MobileVLM extends small-scale transformer models to handle vision-language tasks, making it particularly useful for augmented reality (AR) and real-time visual assistance.

    “MobileVLM, a multimodal variant of TLMs, extends small-scale transformer models to handle vision-language tasks, making it particularly useful for augmented reality (AR) and real-time visual assistance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [114] Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., and Wei, X. (2023). MobileVLM: A Fast, Strong, and Open Vision Language Assistant for Mobile Devices. arXiv.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Industrial robots, conveyor belts, and assembly-line machines can be controlled via spoken instructions, improving accessibility and reducing operational complexity.

    “For example, industrial robots, conveyor belts, and assembly-line machines can be controlled via spoken instructions, improving accessibility and reducing operational complexity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    TLMs help conserve energy by operating on less power-intensive hardware and reducing computational requirements.

    “By operating on less power-intensive hardware and reducing computational requirements, they help conserve energy [72].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [72] Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., and Xu, M. (2024). Small Language Models: Survey, Measurements, and Insights. arXiv.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    TLMs allow IoT gateways to detect anomalies, predict failures, and generate natural language summaries, improving decision-making in industrial workflows.

    “These models allow IoT gateways to detect anomalies, predict failures, and generate natural language summaries, improving decision-making in industrial workflows.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Positional embeddings are crucial for encoding the order of tokens in a sequence in Transformer architectures.

    “Since the Transformer architecture is not inherently sequential, positional embeddings are crucial for encoding the order of tokens in a sequence.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [75] Vaswani, A. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017, Neural Information Processing Systems Foundation, Inc. (NeurIPS).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A key development in TLMs is the shift toward specialized architectures that optimize model efficiency while maintaining performance.

    “A key development in TLMs is the shift toward specialized architectures that optimize model efficiency while maintaining performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Multi-Head Attention improves the Transformer's ability to focus on different parts of the sequence simultaneously.

    “Multi-Head Attention (MHA) [75] is a key component of the Transformer model that improves its ability to focus on different parts of the sequence simultaneously.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [75] Vaswani, A. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017, Neural Information Processing Systems Foundation, Inc. (NeurIPS).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Layer normalization stabilizes training by normalizing inputs within each layer, accelerating convergence.

    “Layer Normalization and Efficiency Enhancements [86] stabilizes the training process by normalizing the inputs within each layer, helping to accelerate convergence.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [86] Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer Normalization. arXiv.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    TLMs can be embedded in devices such as thermostats, security systems, smart lighting, and voice assistants, enabling them to function even in offline or low-connectivity conditions.

    “These models can be embedded in devices such as thermostats, security systems, smart lighting, and voice assistants, enabling them to function even in offline or low-connectivity conditions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    TLMs enable workers to operate machinery using natural language commands, reducing the need for manual input.

    “TLMs enable workers to operate machinery using natural language commands, reducing the need for manual input [148].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [148] Norda, 2023, Evaluating the Efficiency of Voice Control as Human Machine Interface in Production, IEEE Trans. Autom. Sci. Eng. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Low-resource footprint of TLMs facilitates scaling across distributed systems and edge networks, enabling applications such as smart cities and industrial automation.

    “They can be easily scaled across distributed systems and edge networks, enabling applications such as smart cities and industrial automation [48].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [48] Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., and Wang, S. (2024). A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness. arXiv.

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Multi-Query Attention and Grouped Query Attention reduce memory and computational costs while maintaining performance.

    “MQA shares a single set of keys and values across all attention heads, reducing memory usage. GQA takes a middle ground by grouping some query heads together and allowing them to share a set of keys and values. These optimizations reduce memory and computational costs while maintaining performance.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [76] Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head Is All You Need. arXiv.

      Not checked in the free run.

    • cites [77] Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TLMs can deliver faster response times due to their reduced size, which is crucial for real-time applications in fields like healthcare, automation, and conversational AI.

    “Due to their reduced size, TLMs can deliver faster response times, which is crucial for real-time applications in fields like healthcare, automation, and conversational AI [26,73].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [26] Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. (2023). Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. arXiv, Available online: https://arxiv.org/abs/2304.03208.

      Not checked in the free run.

    • cites [73] Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.D.L., Hanna, E.B., and Bress, F. (2024). Mixtral of Experts. arXiv.

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Unlike traditional LLMs, TLMs employ parameter-efficient architectures, reduced-depth transformer layers, and optimized attention mechanisms to achieve competitive accuracy with significantly fewer resources.

    “Unlike traditional LLMs, which rely on large parameter counts and high-dimensional embeddings, TLMs employ parameter-efficient architectures, reduced-depth transformer layers, and optimized attention mechanisms to achieve competitive accuracy with significantly fewer resources.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Multi-Head Latent Attention compresses key-value pairs into a latent vector, significantly reducing overhead while preserving performance.

    “Multi-Head Latent Attention (MLA) [78] takes this a step further by compressing the key-value pairs into a latent vector. This significantly reduces the overhead involved in managing large numbers of keys and values, while still preserving the performance of the model.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [78] Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Deng, C., Ruan, C., Dai, D., and Guo, D. (2024). DeepSeek-v2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Models such as MobileBERT and SqueezeBERT employ bottleneck layers and grouped convolutions, leading to a 3× to 4× reduction in latency while maintaining >90% of the original model's accuracy.

    “Moreover, models such as MobileBERT and SqueezeBERT employ bottleneck layers and grouped convolutions, leading to a 3× to 4× reduction in latency while maintaining >90% of the original model’s accuracy.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [68] Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., and Zhou, D. (2020). MobileBERT: A Compact Task-Agnostic BERT for Resource-Limited Devices. arXiv. source

      Not checked in the free run.

    • cites [103] Iandola, F.N., Shaw, A.E., Krishna, R., and Keutzer, K.W. (2020). SqueezeBERT: What Can Computer Vision Teach NLP About Efficient Neural Networks?. arXiv. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SiLU provides a smooth non-linearity that improves model performance, particularly in deeper models.

    “SiLU combines the sigmoid function with its input, providing a smooth non-linearity that improves model performance, particularly in deeper models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [84] Elfwing, 2018, Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning, Neural Netw. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    These normalization techniques significantly improve training stability, reduce vanishing gradients, and enhance computational efficiency, particularly in small language models.

    “These normalization techniques significantly improve training stability, reduce vanishing gradients, and enhance computational efficiency, particularly in small language models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Rotary Positional Embedding improves the model's ability to handle long-range dependencies and understand token position dynamics.

    “Rotary Positional Embedding (RoPE) [85] enhances this approach by introducing rotational transformations, improving the model’s ability to handle long-range dependencies and better understand the dynamics of token positions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [85] Su, 2024, RoFormer: Enhanced Transformer with Rotary Position Embedding, Neurocomputing source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TLMs are deployed in smart agriculture systems to interpret sensor data on soil moisture, temperature, and humidity, allowing automated irrigation systems to respond to natural language commands.

    “TLMs are deployed in smart agriculture systems to interpret sensor data on soil moisture [146], temperature, and humidity, allowing automated irrigation systems to respond to natural language commands.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [146] Vereecken, 2014, On the Spatio-Temporal Dynamics of Soil Moisture at the Field Scale, J. Hydrol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationown data

    These datasets collectively represent a diverse array of linguistic resources, ranging from general-purpose web corpora to specialized domain-specific datasets, enabling efficient and comprehensive pretraining of TLMs.

    “These datasets collectively represent a diverse array of linguistic resources, ranging from general-purpose web corpora to specialized domain-specific datasets, enabling efficient and comprehensive pretraining of TLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ReLU is a simple activation function widely used for computational efficiency.

    “ReLU is a simple activation function widely used for its computational efficiency, defined as σ(x)=max(0,x).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [82] Agarap, A.F. (2018). Deep Learning Using Rectified Linear Units (ReLU). arXiv.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    RMSNorm simplifies calculation by using the root mean square of inputs instead of standard normalization, reducing computational demands.

    “Additionally, RMS Normalization (RMSNorm) [88] simplifies the calculation by using the root mean square of inputs instead of standard normalization, which reduces computational demands:”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [88] Zhang, B., and Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems, Proceedings of the 2019 Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019, Neural Information Processing Systems Foundation, Inc. (NeurIPS).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The architecture of TLMs is predominantly built upon the foundations established by LLMs, but it is specifically tailored for computational efficiency and scalability.

    “The architecture of TLMs is predominantly built upon the foundations established by LLMs, but it is specifically tailored for computational efficiency and scalability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Parametric Layer Normalization introduces learnable parameters for adaptive scaling and shifting.

    “To enhance model flexibility, Parametric Layer Normalization (PLN) [87] introduces learnable parameters γ and β for adaptive scaling and shifting:”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [87] Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., and Liu, T. (2020, January 13–18). On Layer Normalization in the Transformer Architecture. Proceedings of the 37th International Conference on Machine Learning, PMLR, Virtual.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Flash Attention accelerates attention computation by minimizing memory usage, particularly for longer input sequences.

    “Additionally, Flash Attention [79,80] accelerates the computation of attention by minimizing memory usage, which is particularly important when working with longer input sequences.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [79] Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv.

      Not checked in the free run.

    • cites [80] Dao, 2022, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Adv. Neural Inf. Process. Syst.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GELU offers better gradient flow control and is commonly used in large models like BERT and GPT.

    “GELU is a smoother alternative, defined as GELU(x)=x·Φ(x), where Φ(x) is the standard Gaussian cumulative distribution function (CDF). It offers better gradient flow control and is commonly used in large models like BERT and GPT.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [83] Hendrycks, D., and Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TeenyTinyLlama demonstrates the potential of ultra-compact models for on-device AI assistants, reducing latency by over 60% compared to standard lightweight models.

    “TeenyTinyLlama demonstrates the potential of ultra-compact models for on-device AI assistants, reducing latency by over 60% compared to standard lightweight models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [111] Falk, 2024, TeenyTinyLlama: Open-Source Tiny Language Models Trained in Brazilian Portuguese, Mach. Learn. Appl.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TLMs are capable of processing structured data and converting it into readable, natural language summaries for engineers and operators, improving decision-making and reducing manual reporting efforts.

    “TLMs are capable of processing this structured data and converting it into readable, natural language summaries for engineers and operators, improving decision-making and reducing manual reporting efforts [149].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [149] Xia, Y., Jazdi, N., Zhang, J., Shah, C., and Weyrich, M. (2024). Control Industrial Automation System with Large Language Models. arXiv.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Specialized domain-specific TLMs, such as those designed for scientific, legal, or medical contexts, are paving the way for more targeted, efficient language models that maintain domain relevance while ensuring computational feasibility.

    “Furthermore, specialized domain-specific TLMs, such as those designed for scientific, legal, or medical contexts, are paving the way for more targeted, efficient language models that maintain domain relevance while ensuring computational feasibility.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Their adoption across edge computing, industrial automation, IoT, and healthcare highlights their potential to facilitate real-time, low-power NLP applications.

    “Their adoption across edge computing, industrial automation, IoT, and healthcare highlights their potential to facilitate real-time, low-power NLP applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    TLMs have emerged as a compelling alternative, offering a balance between efficiency and performance.

    “To bridge this gap, TLMs have emerged as a compelling alternative, offering a balance between efficiency and performance.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LLMs demonstrate remarkable capabilities across various applications, but their substantial computational demands and memory footprints hinder their deployment on resource-constrained devices.

  • May combine two claims.

    Premisecitation

    Technicians can input error codes or system alerts, and the model can interpret the issue and suggest corrective actions in natural language.

    “Technicians can input error codes or system alerts, and the model can interpret the issue and suggest corrective actions in natural language [150].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [150] Dhillon, A.S., and Torresin, A. (2024). Advancing Vehicle Diagnostics: Exploring the Application of Large Language Models in the Automotive Industry. [Master’s Thesis, Chalmers University of Technology].

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Workers can ask technical questions, and the model can generate detailed, real-time explanations to assist in problem resolution.

    “Workers can ask technical questions, and the model can generate detailed, real-time explanations to assist in problem resolution.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    TLMs allow engineers to identify potential failures faster and optimize maintenance schedules.

    “This allows engineers to identify potential failures faster and optimize maintenance schedules.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    TLMs face notable challenges, including trade-offs between model size and accuracy, generalization limitations, and ethical considerations related to bias and privacy.

    “Despite their promise, TLMs face notable challenges, including trade-offs between model size and accuracy, generalization limitations, and ethical considerations related to bias and privacy.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Due to their reduced size, TLMs may show limited generalization across diverse tasks or domains without extensive fine-tuning.
    • Rests on an assumption: TLMs often require carefully curated training datasets and advanced training techniques such as knowledge distillation, quantization, or pruning to achieve high performance, which can be resource-intensive, particularly for smaller organizations.
    • Rests on an assumption: Deploying TLMs on edge devices and mobile platforms introduces privacy and security concerns, as models may process sensitive data directly on the device, raising issues around data responsibility and vulnerability to adversarial attacks.
    • Rests on an assumption: Although TLMs are designed for real-time applications, their reduced size can occasionally lead to inconsistent response quality, especially in complex interactions, affecting reliability in critical contexts like healthcare diagnostics or industrial automation.

  • May combine two claims.

    Interpretationreasoning

    TLMs can process these error codes and provide plain-language explanations, helping technicians understand and address issues more effectively.

    “TLMs can process these error codes and provide plain-language explanations, helping technicians understand and address issues more effectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This proactive approach helps industries prevent unexpected breakdowns and schedule maintenance efficiently.

    “This proactive approach helps industries prevent unexpected breakdowns and schedule maintenance efficiently.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    TLMs leverage model compression techniques such as knowledge distillation, quantization, and pruning to retain essential language understanding capabilities while significantly reducing computational overhead.

    “TLMs leverage model compression techniques such as knowledge distillation, quantization, and pruning to retain essential language understanding capabilities while significantly reducing computational overhead.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    By analyzing historical sensor data and real-time system outputs, TLMs can detect early signs of equipment malfunction and predict when components may fail.

    “By analyzing historical sensor data and real-time system outputs, TLMs can detect early signs of equipment malfunction and predict when components may fail [150].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [150] Dhillon, A.S., and Torresin, A. (2024). Advancing Vehicle Diagnostics: Exploring the Application of Large Language Models in the Automotive Industry. [Master’s Thesis, Chalmers University of Technology].

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    TLMs facilitate real-time language translation and multilingual command processing, allowing seamless interaction with industrial systems across diverse linguistic backgrounds.

    “TLMs facilitate real-time language translation and multilingual command processing, allowing seamless interaction with industrial systems across diverse linguistic backgrounds [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., and Yu, D. (2024). MM-LLMs: Recent Advances in Multimodal Large Language Models. arXiv. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    TLMs offer a scalable, efficient, and sustainable alternative tailored for real-time, on-device, and edge-based applications.

    “In summary, while LLMs continue to push the boundaries of artificial intelligence, TLMs offer a scalable, efficient, and sustainable alternative tailored for real-time, on-device, and edge-based applications.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LLMs demonstrate remarkable capabilities across various applications, but their substantial computational demands and memory footprints hinder their deployment on resource-constrained devices.
51 of 51

  • The paper relies on this without support.

    There is a noticeable lack of surveys in the specific area of TLMs, leaving a gap in comprehensive resources.

    “Despite the growing interest and advancements in TLMs, there is a noticeable lack of surveys in this specific area, leaving a gap in comprehensive resources that detail the current state, challenges, and opportunities within the field.”

  • The paper relies on this without support.

    The demand for efficient NLP models that can operate on resource-constrained devices is increasing.

    “The motivation for this paper stems from the increasing demand for efficient NLP models that can operate on resource constrained devices such as mobile phones, IoT systems, and embedded platforms.”

  • The paper relies on this without support.

    Model pruning and quantization are widely used to compress models while preserving performance.

    “Model pruning and quantization, for instance, are widely used to compress models while preserving performance.”

  • The paper relies on this without support.

    Most LLMs are trained predominantly on English corpora.

    “While most LLMs are trained predominantly on English corpora, TLMs are increasingly being fine-tuned for low-resource languages, ensuring broader adoption.”

  • The paper relies on this without support.

    Large-scale generic models often fail to generalize effectively in domain-specific applications.

    “TLMs have demonstrated exceptional flexibility in domain-specific applications, where large-scale generic models often fail to generalize effectively.”

  • The paper relies on this without support.

    Conventional LLMs remain impractical on specialized hardware such as microcontrollers, edge TPU accelerators, and mobile processors.

    “Their efficiency allows them to be deployed on specialized hardware, such as microcontrollers, edge TPU accelerators, and mobile processors, where conventional LLMs remain impractical.”

  • The paper relies on this without support.

    Due to their reduced size, TLMs may show limited generalization across diverse tasks or domains without extensive fine-tuning.

    “Due to their reduced size, TLMs may show limited generalization across diverse tasks or domains without extensive fine-tuning, restricting their usability in applications requiring versatile language understanding and adaptability.”

  • The paper relies on this without support.

    TLMs often require carefully curated training datasets and advanced training techniques such as knowledge distillation, quantization, or pruning to achieve high performance, which can be resource-intensive, particularly for smaller organizations.

    “TLMs often require carefully curated training datasets and advanced training techniques such as knowledge distillation, quantization, or pruning to achieve high performance, which can be resource-intensive, particularly for smaller organizations.”

  • The paper relies on this without support.

    Deploying TLMs on edge devices and mobile platforms introduces privacy and security concerns, as models may process sensitive data directly on the device, raising issues around data responsibility and vulnerability to adversarial attacks.

    “Deploying TLMs on edge devices and mobile platforms introduces privacy and security concerns, as models may process sensitive data directly on the device, raising issues around data responsibility and vulnerability to adversarial attacks.”

  • The paper relies on this without support.

    Although TLMs are designed for real-time applications, their reduced size can occasionally lead to inconsistent response quality, especially in complex interactions, affecting reliability in critical contexts like healthcare diagnostics or industrial automation.

    “Although TLMs are designed for real-time applications, their reduced size can occasionally lead to inconsistent response quality, especially in complex interactions, affecting reliability in critical contexts like healthcare diagnostics or industrial automation.”

  • The paper relies on this without support.

    LLMs demonstrate remarkable capabilities across various applications, but their substantial computational demands and memory footprints hinder their deployment on resource-constrained devices.

    “While these models demonstrate remarkable capabilities across various applications, their substantial computational demands and memory footprints hinder their deployment on resource-constrained devices.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Offline knowledge distillation offers training stability and modularity, allowing the teacher to fully converge using richer multimodal signals before transferring knowledge to the unimodal student.

    “First, offline distillation offers training stability and modularity, allowing the teacher to fully converge using richer multimodal signals (e.g., inertial and skeleton) before transferring knowledge to the unimodal student.”

    From A tiny inertial transformer for human activity recognition via multimodal knowledge distillation and explainable AI (2025) · cites this paper as [45]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • TinyLlama is dated both 2024 and 2023

    Minor

    The introduction places TinyLlama among models introduced in 2024, but Table 3 dates TinyLlama to 2023.

  • Cerebras-GPT table citation points to MobileVLM

    Minor

    Table 3 cites reference 114 for Cerebras-GPT, but the paper's reference 114 is titled MobileVLM. The Cerebras-GPT discussion instead cites reference 26.

  • Conflicting MobileBERT size reductions

    Minor

    The same description calls MobileBERT about 50% smaller and 4.3× smaller than BERTBASE. A 4.3-fold size reduction leaves about 23% of the original size, rather than about 50%.

3 of 3

Cites

Flagged first, then by citations

14 of the 113 references couldn’t be matched to a known paper.

Research works printed but not linked: 57

  1. Zhao, W.X., Qin, J., Wang, Y., Liu, Y., Yang, Z., and Gao, Z. (2023). A Survey of Large Language Models. arXiv.
  2. Naveed, H., Ali, A., Rehman, K., and Hashmi, M.A. (2023). A Comprehensive Overview of Large Language Models. arXiv.
  3. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv.
  4. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and Efficient Foundation Language Models. arXiv.
  5. Bhat, M.M., Meng, R., Liu, Y., Zhou, Y., and Yavuz, S. (2023). Investigating Answerability of LLMs for Long-Form Question Answering. arXiv.
  6. Bai, G., Chai, Z., Ling, C., Wang, S., Lu, J., Zhang, N., Shi, T., Yu, Z., Zhu, M., and Zhang, Y. (2024). Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models. arXiv.
  7. Han, S., Mao, H., and Dally, W.J. (2015). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv.
  8. Tang, Y., Liu, F., Ni, Y., Tian, Y., Bai, Z., Hu, Y.Q., Liu, S., Jui, S., Han, K., and Wang, Y. (2024). Rethinking Optimization and Architecture for Tiny Language Models. arXiv.
  9. Zhang, D., Hu, Z., Zhoubian, S., Du, Z., Yang, K., Wang, Z., and Tang, J. (2024, January 10–15). SciInstruct: A Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models. Proceedings of the Thirty-Eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Vancouver, BC, Canada.
  10. Zhang, D., Liu, W., Tan, Q., Chen, J., Yan, H., Yan, Y., and Ouyang, W. (2024). ChemLLM: A Chemical Large Language Model. arXiv.
  11. Mehta, S., Sekhavat, M.H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, S.I., Najibi, M., Belenko, D., and Zatloukal, P. (2024, January 26). OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. Proceedings of the Workshop on Efficient Systems for Foundation Models II@ICML2024, Vienna, Austria.
  12. Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., and Wang, S. (2024). A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness. arXiv.
  13. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
  14. Gu, Y., Dong, L., Wei, F., and Huang, M. (2023). Knowledge Distillation of Large Language Models. arXiv.
  15. Gu, Y., Dong, L., Wei, F., and Huang, M. (2024, January 7–11). MiniLLM: Knowledge Distillation of Large Language Models. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.
  16. Cho, J.H., and Hariharan, B. (November, January 27). On the Efficacy of Knowledge Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
  17. Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q., Wang, Y., and Mahoney, M. (2021, January 18–24). Hawq-v3: Dyadic Neural Network Quantization. Proceedings of the International Conference on Machine Learning, Virtual.
  18. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized Neural Networks. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2016, Neural Information Processing Systems Foundation, Inc. (NeurIPS).
  19. Liu, P., Liu, Z., Gao, Z.F., Gao, D., Zhao, W.X., Li, Y., Ding, B., and Wen, J.R. (2023). Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study. arXiv.
  20. Lee, N., Ajanthan, T., and Torr, P.H. (2018). Snip: Single-Shot Network Pruning Based on Connection Sensitivity. arXiv.
  21. Dery, L., Kolawole, S., Kagy, J.F., Smith, V., Neubig, G., and Talwalkar, A. (2024). Everybody Prune Now: Structured Pruning of LLMs with Only Forward Passes. arXiv.
  22. Hillier, D., Guertler, L., Tan, C., Agrawal, P., Ruirui, R.C., and Cheng, B. (2024). Super Tiny Language Models. arXiv.
  23. Mitchell, E., Rafailov, R., Sharma, A., Finn, C., and Manning, C.D. (2023). An Emulator for Fine-Tuning Large Language Models Using Small Language Models. arXiv.
  24. Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., and Xu, M. (2024). Small Language Models: Survey, Measurements, and Insights. arXiv.
  25. Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.D.L., Hanna, E.B., and Bress, F. (2024). Mixtral of Experts. arXiv.
  26. Vaswani, A. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017, Neural Information Processing Systems Foundation, Inc. (NeurIPS).
  27. Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head Is All You Need. arXiv.
  28. Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Deng, C., Ruan, C., Dai, D., and Guo, D. (2024). DeepSeek-v2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv.
  29. Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv.
  30. Agarap, A.F. (2018). Deep Learning Using Rectified Linear Units (ReLU). arXiv.
  31. Hendrycks, D., and Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv.
  32. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer Normalization. arXiv.
  33. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., and Liu, T. (2020, January 13–18). On Layer Normalization in the Transformer Architecture. Proceedings of the 37th International Conference on Machine Learning, PMLR, Virtual.
  34. Zhang, B., and Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems, Proceedings of the 2019 Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019, Neural Information Processing Systems Foundation, Inc. (NeurIPS).
  35. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., and Launay, J. (2023). The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv.
  36. Nguyen, T., Van Nguyen, C., Lai, V.D., Man, H., Ngo, N.T., Dernoncourt, F., and Nguyen, T.H. (2023). CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages. arXiv.
  37. Penedo, G., Kydlíček, H., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv.
  38. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., and Nabeshima, N. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv.
  39. Weber, M., Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., and Zhang, C. (2024). RedPajama: An Open Dataset for Training Large Language Models. arXiv.
  40. Liu, Y. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
  41. Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., and Shankar, V. (2024). DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. arXiv.
  42. Li, R., Ben Allal, L., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., and de Vries, H. (2023). StarCoder: May the Source Be With You!. arXiv.
  43. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
  44. Thawakar, O., Vayani, A., Khan, S., Cholakal, H., Anwer, R.M., Felsberg, M., Baldwin, T., Xing, E.P., and Khan, F.S. (2024). MobilLlama: Towards Accurate and Lightweight Fully Transparent GPT. arXiv.
  45. Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravinskyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., and Datta, A. (2024). Stable LM 2 1.6B Technical Report. arXiv.
  46. Mitra, A., Del Corro, L., Mahajan, S., Codas, A., Simoes, C., Agarwal, S., Chen, X., Razdaibiedina, A., Jones, E., and Aggarwal, K. (2023). Orca 2: Teaching Small Language Models How to Reason. arXiv.
  47. Galanos, T., Liapis, A., and Yannakakis, G.N. (2023). Architext: Language-Driven Generative Architecture Design. arXiv.
  48. Allal, L.B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C.M., Muennighoff, N., Mishra, M., Gu, A., and Dey, M. (2023). SantaCoder: Don’t Reach for the Stars!. arXiv.
  49. Du, X., Yu, Z., Gao, S., Pan, D., Cheng, Y., Ma, Z., Yuan, R., Qu, X., Liu, J., and Zheng, T. (2024). Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model. arXiv.
  50. Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., and Wei, X. (2023). MobileVLM: A Fast, Strong, and Open Vision Language Assistant for Mobile Devices. arXiv.
  51. Yuan, Z., Li, Z., Huang, W., Ye, Y., and Sun, L. (2023). TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones. arXiv.
  52. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., and Huang, F. (2024). Qwen2 Technical Report. arXiv.
  53. Franch, G., Tomasi, E., Wanjari, R., Poli, V., Cardinali, C., Alberoni, P.P., and Cristoforetti, M. (2024). GPTCast: A Weather Language Model for Precipitation Nowcasting. arXiv.
  54. Xia, Y., Jazdi, N., Zhang, J., Shah, C., and Weyrich, M. (2024). Control Industrial Automation System with Large Language Models. arXiv.
  55. Dhillon, A.S., and Torresin, A. (2024). Advancing Vehicle Diagnostics: Exploring the Application of Large Language Models in the Automotive Industry. [Master’s Thesis, Chalmers University of Technology].
  56. Ouali, Y., Hudelot, C., and Tami, M. (2020). An Overview of Deep Semi-Supervised Learning. arXiv.
  57. Wei, H., Zheng, G., Gayah, V., and Li, Z. (2019). A Survey on Traffic Signal Control Methods. arXiv.

Cited by

Sorted by citations

Related

Authors
Lamaakal, I., Maleh, Y., El Makkaoui, K., Ouahbi, I., Pławiak, P., Alfarraj, O., Almousa, M., & Abd El-Latif, A. A.
Affiliations
  • Mohamed I University (MA)
  • Université Sultan Moulay Slimane (MA)
  • Multidisciplinary Faculty of Nador, Mohammed Premier University, Oujda 60000, Morocco
  • Cracow University of Technology (PL)
  • King Saud University (SA)
  • Princess Nourah bint Abdulrahman University (SA)
  • EIAS Data Science Lab, Center of Excellence in Quantum and Intelligent Computing, College of Computer and Information Sciences, Prince Sultan University, Riyadh 11586, Saudi Arabia
Corresponding author
Maleh, Yassine
ORCID
Venue
Sensors
Publisher
MDPI AG
Published
2025-02-21
Volume
25
Issue
5
Pages
1318
ISSN
1424-8220 (ISSN-L)
DOI
10.3390/s25051318
PMC
PMC11902656
OpenAlex
W4407809258
Semantic Scholar
174828b817d0d0671e2b9a368fb01b17971487c5
Cited by
60
References
113
Citations by year
2026: 34 · 2025: 26
FWCI
21.07
Type
journal article
Language
en
Open access
Licence
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • Princess Nourah bint Abdulrahman University Researchers Supporting (PNURSP2025R752)
  • Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia (PNURSP2025R752)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
Keywords
BabyLM, model compression, knowledge distillation, quantization, pruning, edge computing, Internet of Things, industrial automation, medical applications, resource-constrained devices, context-aware models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review