CoChat
Sign Up
Cited by 9Reinforcement Learning in RoboticsIoT and Edge/Fog ComputingDigital Transformation in Industry

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Resource-Efficient Sensor Data Management for Autonomous Systems Using Deep Reinforcement Learning

Open access
Jeong, Seunghwan, Yoo, Gwangpyo, Yoo, Minjong, Yeom, Ikjun, Woo, Honguk·2019·Sensors
DOI 10.3390/s19204410PMC PMC6832860
Open this paper in CoChat — free accountOpen PDF

Abstract

Hyperconnectivity via modern Internet of Things (IoT) technologies has recently driven us to envision "digital twin", in which physical attributes are all embedded, and their latest updates are synchronized on digital spaces in a timely fashion. From the point of view of cyberphysical system (CPS) architectures, the goals of digital twin include providing common programming abstraction on the same level of databases, thereby facilitating seamless integration of real-world physical objects and digital assets at several different system layers. However, the inherent limitations of sampling and observing physical attributes often pose issues related to data uncertainty in practice. In this paper, we propose a learning-based data management scheme where the implementation is layered between sensors attached to physical attributes and domain-specific applications, thereby mitigating the data uncertainty between them. To do so, we present a sensor data management framework, namely D2WIN, which adopts reinforcement learning (RL) techniques to manage the data quality for CPS applications and autonomous systems. To deal with the scale issue incurred by many physical attributes and sensor streams when adopting RL, we propose an action embedding strategy that exploits their distance-based similarity in the physical space coordination. We introduce two embedding methods, i.e., a user-defined function and a generative model, for different conditions. Through experiments, we demonstrate that the D2WIN framework with the action embedding outperforms several known heuristics in terms of achievable data quality under certain resource restrictions. We also test the framework with an autonomous driving simulator, clearly showing its benefit. For example, with only 30% of updates selectively applied by the learned policy, the driving agent maintains its performance about 96.2%, as compared to the ideal condition with full updates.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/s19204410

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Effects of resource restriction: The driving test score of the simulator [6] with respect to (a) resource utilization limits for sensor data updates in percentages, where higher limits indicate more resource usages (on the X-axis); (b) average time interval in frames for consecutive updates of sensor data where large intervals indicate less resource usages; and (c) average time delay in frames for consecutive updates of sensor data where longer delays indicate less resource usage.
Figure 1 Effects of resource restriction: The driving test score of the simulator [6] with respect to (a) resource utilization limits for sensor data updates in percentages, where higher limits indicate more resource usages (on the X-axis); (b) average time interval in frames for consecutive updates of sensor data where large intervals indicate less resource usages; and (c) average time delay in frames for consecutive updates of sensor data where longer delays indicate less resource usage.CC BY · PubMed Central
Figure 2 Effects of resource restriction: Observation examples in the case of Figure 1a where the numbers on the line denote the resource limits.
Figure 2 Effects of resource restriction: Observation examples in the case of Figure 1a where the numbers on the line denote the resource limits.CC BY · PubMed Central
Figure 3 Conceptual system architecture for sensor data management: The D2WIN framework acts as a mediator between a domain-specific, data-intensive application, and a network system with multiple sensors.
Figure 3 Conceptual system architecture for sensor data management: The D2WIN framework acts as a mediator between a domain-specific, data-intensive application, and a network system with multiple sensors.CC BY · PubMed Central
Figure 4 Framework architecture: The RL-driven multistage tasks (1–7) used to schedule the jobs for updating objects are described with the respective numbers on the control and data flows.
Figure 4 Framework architecture: The RL-driven multistage tasks (1–7) used to schedule the jobs for updating objects are described with the respective numbers on the control and data flows.CC BY · PubMed Central
Figure 5 LA3 architecture: In the orchestrator (ORC), locality-aware action abstraction (LA3) treats the object-wise scheduling task as a two-step procedure (as illustrated in the left-box of diagram). The second step (T(·)|ρ) is implemented based on either the user-defined function or the generative model-based function.
Figure 5 LA3 architecture: In the orchestrator (ORC), locality-aware action abstraction (LA3) treats the object-wise scheduling task as a two-step procedure (as illustrated in the left-box of diagram). The second step (T(·)|ρ) is implemented based on either the user-defined function or the generative model-based function.CC BY · PubMed Central
Figure 6 Branch diagram for the derivative of J.
Figure 6 Branch diagram for the derivative of J.CC BY · PubMed Central
Figure 7 LA3 with the generative model: (a) The model is first trained to learn the features of N-scores from the trace data for relatively large N. Its decoder is then placed for LA3’s action transformation (b). As a result, the action space size of the RL agent becomes |ρ|≪N.
Figure 7 LA3 with the generative model: (a) The model is first trained to learn the features of N-scores from the trace data for relatively large N. Its decoder is then placed for LA3’s action transformation (b). As a result, the action space size of the RL agent becomes |ρ|≪N.CC BY · PubMed Central
Figure 8 Object-wise reinforcement learning (RL) curve over time.
Figure 8 Object-wise reinforcement learning (RL) curve over time.CC BY · PubMed Central
Figure 9 LA3 learning curve over time.
Figure 9 LA3 learning curve over time.CC BY · PubMed Central
Figure 10 VOL scale.
Figure 10 VOL scale.CC BY · PubMed Central
Figure 11 Resource limit.
Figure 11 Resource limit.CC BY · PubMed Central
Figure 12 Query arrival time.
Figure 12 Query arrival time.CC BY · PubMed Central
Figure 13 Query dynamics.
Figure 13 Query dynamics.CC BY · PubMed Central
Figure 14 Object staleness snapshot examples by the models in Table 1.
Figure 14 Object staleness snapshot examples by the models in Table 1.CC BY · PubMed Central
Figure 15 Performance of the generative model for action transformation: LA3-G and LA3 share the same structure for two-step action embedding, except that LA3-G exploits the generative model in Figure 7, but LA3 exploits the user-defined function.
Figure 15 Performance of the generative model for action transformation: LA3-G and LA3 share the same structure for two-step action embedding, except that LA3-G exploits the generative model in Figure 7, but LA3 exploits the user-defined function.CC BY · PubMed Central
Figure 16 Performances test with a state-of-the-art variant.
Figure 16 Performances test with a state-of-the-art variant.CC BY · PubMed Central
Figure 17 Case study system architecture: for this simulation test, we implemented two additional modules, i.e., query emulator and object-image mapper, in addition to the generic framework explained previously. The two modules are denoted by the gray box. Importantly, the implementation and integration of the two modules requires no modification in the simulator (which emulates a system environment with many sensors) or the driving agent (which emulates a domain specific application in Figure 4).
Figure 17 Case study system architecture: for this simulation test, we implemented two additional modules, i.e., query emulator and object-image mapper, in addition to the generic framework explained previously. The two modules are denoted by the gray box. Importantly, the implementation and integration of the two modules requires no modification in the simulator (which emulates a system environment with many sensors) or the driving agent (which emulates a domain specific application in Figure 4).CC BY · PubMed Central
Figure 18 Internal data examples in the case study system: the examples (a–c) containing the system state are captured in the framework, example (d) is captured internally in the simulator, and example (e) is in the object-image mapper. Note that the full observation (d) is only available inside the simulator, and is not used by the framework or the driving agent; it is presented here only for comparison purposes.
Figure 18 Internal data examples in the case study system: the examples (a–c) containing the system state are captured in the framework, example (d) is captured internally in the simulator, and example (e) is in the object-image mapper. Note that the full observation (d) is only available inside the simulator, and is not used by the framework or the driving agent; it is presented here only for comparison purposes.CC BY · PubMed Central
Figure 19 Performance evaluation: (a) we configure the resource limit in percentages at several stages (on the X-axis) and denote the driving score; (b) we configure the dimension of abstract actions |ρ| (on the X-axis). (b) For the VOL scale of 40², we observe stable performance highly better than SOF with |ρ|=8²,16², but does not with |ρ|=2²,4²,32².
Figure 19 Performance evaluation: (a) we configure the resource limit in percentages at several stages (on the X-axis) and denote the driving score; (b) we configure the dimension of abstract actions |ρ| (on the X-axis). (b) For the VOL scale of 40², we observe stable performance highly better than SOF with |ρ|=8²,16², but does not with |ρ|=2²,4²,32².CC BY · PubMed Central
Figure 20 Comparison of stable and unstable cases: (a–c) the learning curves for the cases of |ρ|=2², 16², and 32², respectively, in Figure 19b, illustrating effects by |ρ| value settings, the dimension of abstract actions.
Figure 20 Comparison of stable and unstable cases: (a–c) the learning curves for the cases of |ρ|=2², 16², and 32², respectively, in Figure 19b, illustrating effects by |ρ| value settings, the dimension of abstract actions.CC BY · PubMed Central
Figure 21 Variant objective test: LA3 for this test aims at minimizing the number of sensor data updates (so, reducing the resource utilization (a)) while satisfying a given QoD requirement (e.g., 95% (b)).
Figure 21 Variant objective test: LA3 for this test aims at minimizing the number of sensor data updates (so, reducing the resource utilization (a)) while satisfying a given QoD requirement (e.g., 95% (b)).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 30

  1. Bradley, 2015, Optimization and Control of Cyberphysical Vehicle Systems, Sensors, 15, 23020, 10.3390/s150923020 · DOI
  2. Uhlemann, 2017, The Digital Twin: Realizing the Cyber-Physical Production System for Industry 4.0, Procedia CIRP, 61, 335, 10.1016/j.procir.2016.11.152 · DOI · Paper page
  3. Uhlemann, 2017, The Digital Twin: Demonstrating the Potential of Real Time Data Acquisition in Production Systems, Procedia Manuf., 9, 113, 10.1016/j.promfg.2017.04.043 · DOI
  4. Deshpande, A., Guestrin, C., Madden, S.R., Hellerstein, J.M., and Hong, W. (September, January 31). Model-driven Data Acquisition in Sensor Networks. Proceedings of the Thirtieth International Conference on Very Large Data Bases, Toronto, ON, Canada.
  5. Deshpande, A., and Madden, S. (2006;, January 27–29). MauveDB: supporting model-based user views in database systems. Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, Chicago, IL, USA. · DOI
  6. Morison, A.M., Murphy, T., and Woods, D.D. (2016, January 17–22). Seeing Through Multiple Sensors into Distant Scenes: The Essential Power of Viewpoint Control. Proceedings of the International Conference Human-Computer Interaction Platforms and Techniques, Toronto, ON, Canada. · DOI
  7. Peng, 2015, Random task scheduling scheme based on reinforcement learning in cloud computing, Cluster Comput., 18, 1595, 10.1007/s10586-015-0484-2 · DOI
  8. Bao, Y., Peng, Y., and Wu, C. (May, January 29). Deep Learning-based Job Placement in Distributed Machine Learning Clusters. Proceedings of the IEEE Conference on Computer Communications, Paris, France.
  9. Mao, H., Schwarzkopf, M., Venkatakrishnan, S.B., Meng, Z., and Alizadeh, M. (2019, January 19–24). Learning Scheduling Algorithms for Data Processing Clusters. Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM), Beijing, China. · DOI
  10. Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J.M., and Stoica, I. (2018). Learning to Optimize Join Queries with Deep Reinforcement Learning. arXiv.
  11. Kang, 2004, Managing deadline miss ratio and sensor data freshness in real-time databases, IEEE Trans. Knowl. Data Eng., 16, 1200, 10.1109/TKDE.2004.61 · DOI
  12. Zhou, Y., and Kang, K.D. (2010, January 6–9). Deadline assignment and tardiness control for real-time data services. Proceedings of the 2010 22nd Euromicro Conference on Real-Time Systems, Brussels, Belgium. · DOI
  13. Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press. · DOI · Paper page
  14. Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv.
  15. Bartle, R.G., and Bartle, R.G. (1995). The Elements of Integration and Lebesgue Measure, A Wiley-Interscience. · DOI
  16. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv.
  17. Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016, January 19–24). Benchmarking deep reinforcement learning for continuous control. Proceedings of the 33rd International International Conference on Machine Learning (ICML), New York, NY, USA.
  18. Ramamritham, 1993, Real-time databases, Distrib. Parallel Databases, 1, 199, 10.1007/BF01264051 · DOI
  19. Zhou, 2015, Deadline Assignment and Feedback Control for Differentiated Real-Time Data Services, IEEE Trans. Knowl. Data Eng., 27, 3245, 10.1109/TKDE.2015.2441725 · DOI
  20. Mnih, V., Heess, N., Graves, A., and Kavukcuoglu, K. (2014, January 8–13). Recurrent models of visual attention. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
  21. Elgabli, A., Khan, H., Krouka, M., and Bennis, M. (2018). Reinforcement Learning Based Scheduling Algorithm for Optimizing Age of Information in Ultra Reliable Low Latency Networks. arXiv. · DOI
  22. Chowdhury, 2019, DA-DRLS: Drift adaptive deep reinforcement learning based scheduling for IoT resource management, J. Netw. Comput. Appl., 138, 51, 10.1016/j.jnca.2019.04.010 · DOI
  23. Pazis, J., and Parr, R. (July, January 28). Generalized value functions for large action sets. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA.
  24. Sallab, 2017, Deep reinforcement learning framework for autonomous driving, Electron. Imaging, 2017, 70, 10.2352/ISSN.2470-1173.2017.19.AVM-023 · DOI · Paper page
  25. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 19–24). Asynchronous methods for deep reinforcement learning. Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA.
  26. Li, 2019, Reinforcement Learning and Deep Learning Based Lateral Control for Autonomous Driving [Application Notes], IEEE Comput. Intell. Mag., 14, 83, 10.1109/MCI.2019.2901089 · DOI
  27. Kaushik, M., Prasad, V., Krishna, K.M., and Ravindran, B. (2018, January 26–30). Overtaking maneuvers in simulated highway driving using deep reinforcement learning. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China. · DOI
  28. Wu, K., Abolfazli Esfahani, M., Yuan, S., and Wang, H. (2018). Learn to steer through deep reinforcement learning. Sensors, 18. · DOI
  29. Zhou, X., Gao, Y., and Guan, L. (2019). Towards goal-directed navigation through combining learning based global and local planners. Sensors, 19. · DOI
  30. Jaritz, M., de Charette, R., Toromanoff, M., Perot, E., and Nashashibi, F. (2018, January 21–26). End-to-end race driving with deep reinforcement learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia. · DOI · Paper page

Cited by

Sorted by citations

See all 9

Metadata

Authors
Jeong, S., Yoo, G., Yoo, M., Yeom, I., & Woo, H.
Affiliations
Sungkyunkwan University (KR)
Corresponding author
Woo, Honguk
ORCID
Venue
Sensors
Publisher
MDPI AG
Published
2019-10-11
Volume
19
Issue
20
Pages
4410
ISSN
1424-8220 (ISSN-L)
DOI
10.3390/s19204410
PMC
PMC6832860
OpenAlex
W2979953627
Cited by
9
References
5
Citations by year
2025: 1 · 2024: 1 · 2023: 3 · 2022: 1 · 2021: 2 · 2020: 1
FWCI
0.46
Type
journal article
Language
en
Open access
Licence
Locations
  • Sensors (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Research Foundation of Korea (NRF-2016R1E1A1A01943474, NRF-2018R1D1A1A02086102)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
  • Physical Sciences › Engineering › Industrial and Manufacturing Engineering › Digital Transformation in Industry
Keywords
sensor data management, digital twin, cyber-physical systems, deep reinforcement learning, data quality, data uncertainty, generative models, autonomous driving, Internet of Things, resource efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Resource-Efficient Sensor Data Management for Autonomous Systems Using Deep Reinforcement Learning. https://app.cochat.ai/papers/10.3390/s19204410
The paper
Jeong, S., Yoo, G., Yoo, M., Yeom, I., & Woo, H. (2019). Resource-Efficient Sensor Data Management for Autonomous Systems Using Deep Reinforcement Learning. Sensors. https://doi.org/10.3390/s19204410