A 5-Step Framework for Measuring the Quality of Experience (QoE) of Internet of Things (IoT) Services
Open accessAbstract
The ultimate benefit of the Internet of Things (IoT) is to improve quality of life by supporting the automation and interconnection of various services; this interconnection includes public facilities in platforms such as smart cities. IoT implementation is believed to improve services and yield additional benefits. However, assessment of how the IoT brings such an experience to users is still a challenge that is not yet standardized. Given that little research has discussed this issue, this paper aims to formulate a framework for measuring the quality of experience (QoE) regarding IoT services on the basis of the Absolute Category Rating with Hidden Reference (ACR-HR) scale, which measures the perception of experience by having the IoT users provide scores for their experiences before and after IoT implementation. The proposed framework consists of 5 steps: setting up the focus of the IoT services and the QoE parameters being assessed, defining users of IoT services, conducting a mean opinion score (MOS) survey of the users, calculating the DMOS (differential MOS) on the basis of the ACR-HR quantitative scale, and finally providing the strategic implications to those who implement the service. To test the proposed framework, we have conducted a proof of concept by measuring the QoE for the IoT services in the Jakarta Smart City platform. Six institutional users operate the IoT for their public services, ranging from fire and rescue brigades to public bus transportation. The results imply that the IoT-based services provide a good experience to institutional users when compared to the state of services without IoT. A strategic implication is that some features of Jakarta Smart City must be evaluated and improved to increase the level of satisfaction of users. To conclude, our framework is sufficiently reliable in measuring the QoE for IoT-based services and can be recommended for use for any other IoT-type service.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some other existing literature [30,37,38] do not assess how the application QoE relates to the individual components of IoT applications.
“The same is true with some of the other existing literature [30,37,38], as they do not assess how the application QoE relates to the individual components of IoT applications.”
From QoE in IoT: a vision, survey and future directions (2021) · cites this paper as [38]
They model the application QoE and measure the users experience through subjective evaluations.
“They model the application QoE and measure the users experience through subjective evaluations.”
From QoE in IoT: a vision, survey and future directions (2021) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035 · see citing works
Foundations for Smarter Cities
Harrison, C., Eckman, B. A., Hamilton, R. W., Hartswick, P., Kalagnanam, J., Paraszczak, J. R., & Williams, P. (2010) · IBM Journal of Research and Development · doi:10.1147/jrd.2010.2048257 · Cited by 1,355 · see citing works
The internet of things: mapping the value beyond the hype
Manyika, J., Chui, M., Bisson, P., Woetzel, J. R., Dobbs, R., Bughin, J., & Aharon, D. (2015) · Cited by 788 · see citing works
Internet of Things (IoT) for Smart Precision Agriculture and Farming in Rural Areas
Ahmed, N., De, D., & Hussain, M. I. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2879579 · Cited by 634 · see citing works
Enabling the Internet of Things
Want, R., Schilit, B. N., & Jenson, S. (2015) · Computer · doi:10.1109/mc.2015.12 · Cited by 444 · see citing works
Modeling Citizen Satisfaction with Mandatory Adoption of an E-Government Technology
Chan, F., Thong, J. Y., Venkatesh, V., Brown, S. A., Hu, P. J.-H., & Tam, K. (2010) · Journal of the AIS · doi:10.17705/1jais.00239 · Cited by 429 · see citing works
How does immersion work in augmented reality games? A user-centric view of immersion and engagement
Shin, D. (2017) · Information Communication & Society · doi:10.1080/1369118x.2017.1411519 · Cited by 235 · see citing works
Shin, D. (2017) · Information & Management · doi:10.1016/j.im.2017.02.006 · Cited by 206 · see citing works
Firmansyah, H. S., Supangkat, S. H., Arman, A. A., & Giabbanelli, P. J. (2019) · IEEE Access · doi:10.1109/access.2019.2908622 · Cited by 85 · see citing works
Toward Understanding Crowd Mobility in Smart Cities through the Internet of Things
Solmaz, G., Wu, F., Cirillo, F., et al. (2019) · IEEE Communications Magazine · doi:10.1109/mcom.2019.1800611 · Cited by 68 · see citing works
Integrated acceptance and sustainability evaluation of Internet of Medical Things
Shin, D., & Hwang, Y. (2017) · Internet Research · doi:10.1108/intr-07-2016-0200 · Cited by 60 · see citing works
Managing the Quality of Experience in the Multimedia Internet of Things: A Layered-Based Approach
Floris, A., & Atzori, L. (2016) · Sensors · doi:10.3390/s16122057 · Cited by 54 · cited 1× · see citing works
“In a more specific sector, we note reference [24], which proposed a conceptual model for QoE in the case of the Internet of Medical Things (IoMT); reference [25], which proposed a user experience model in the case of an augmented reality (AR) game; and reference [26], which proposed a QoE layered management framework…”
BackgroundA Quality Model for Evaluating IoT Applications
Kim, M. (2016) · International Journal of Computer and Electrical Engineering · doi:10.17706/ijcee.2016.8.1.66-76 · Cited by 53 · see citing works
Data-Driven QoS and QoE Management in Smart Cities: A Tutorial Study
Tu, W. (2018) · IEEE Communications Magazine · doi:10.1109/mcom.2018.1700870 · Cited by 47 · see citing works
Dlamini, N. N., & Johnston, K. A. (2016) · International Conference on Advances in Computing and Communication Engineering · doi:10.1109/icacce.2016.8073787 · Cited by 44 · see citing works
A framework of scalable QoE modeling for application explosion in the Internet of Things
Ikeda, Y., Kouno, S., Shiozu, A., & Noritake, K. (2016) · 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT) · doi:10.1109/wf-iot.2016.7845393 · Cited by 25 · see citing works
Suryanegara, M., Arifin, A. S., Asvial, M., et al. (2018) · IEEE Access · doi:10.1109/access.2018.2885375 · Cited by 25 · see citing works
Self-citationDevelopment of smart public transportation system in Jakarta city based on integrated IoT platform
Murad, D. F., Abbas, B. S., Trisetyarso, A., et al. (2018) · International Conference on Information and Communications Technology (ICOIACT) · doi:10.1109/icoiact.2018.8350812 · Cited by 21 · see citing works
The system engineering process
Boarder, J. C. (2002) · doi:10.1109/iemc.1995.524596 · Cited by 19 · see citing works
INNOVATION MANAGEMENT: LESSONS LEARNED FROM INNOVATION DIAGNOSTIC TOOLS
Bagno, R. B., Leiva, T. L., & de Oliveira, L. G. H. (2016) · Product Management & Development · doi:10.4322/pmd.2016.005 · Cited by 13 · see citing works
What Changes After Switching to 4G-LTE? Findings From the Indonesian Market
Suryanegara, M., Andriyanto, F., & Winarko, B. (2017) · IEEE Access · doi:10.1109/access.2017.2736012 · Cited by 13 · see citing works
Self-citationQuality of Experience (QoE) in the smart cities context: An initial analysis
Ballesteros, L. G. M., Alvarez, O., & Markendahl, J. I. (2015) · IEEE International Smart Cities Conference (ISC2) · doi:10.1109/isc2.2015.7366222 · Cited by 12 · see citing works
Spectrum requirement for IoT services: A case of Jakarta smart city
Kusumawati, D., Setiawan, D., & Suryanegara, M. (2017) · IEEE International Conference on Communication, Networks and Satellite (COMNETSAT) · doi:10.1109/comnetsat.2017.8263567 · Cited by 12 · see citing works
Self-citationIoT-based defect predictive manufacturing systems
Kwon, Y. J., & Kim, D. H. (2017) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc.2017.8190856 · Cited by 9 · see citing works
The Role of Living Labs in Developing Smart Cities in Indonesia
Supangkat, S. H., Arman, A. A., Fatimah, Y. A., et al. (2019) · S.M.A.R.T. environments · doi:10.1007/978-3-030-12180-8_11 · Cited by 7 · see citing works
Lessons Learned from the Quality of Experience (QoE) Assessment of 4G Mobile Technology in Indonesia
Suryanegara, M., Andriyanto, F., & Arifin, A. S. (2018) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v10.i3.pp1203-1211 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Ericsson Mobility Report, Special Edition World Economic Forum, Erics- son, Stockholm, Sweden, 2019.
Cited by
Sorted by citations
QoE in IoT: a vision, survey and future directions
Fizza, K., Banerjee, A., Mitra, K., Jayaraman, P. P., Ranjan, R., Patel, P., & Georgakopoulos, D. (2021) · Discover Internet of Things · doi:10.1007/s43926-021-00006-7 · Cited by 107 · see citing works
- Authors
- Suryanegara, M., Prasetyo, D. A., Andriyanto, F., & Hayati, N.
- Affiliations
- University of Indonesia (ID)
- Muhammadiyah University of Yogyakarta (ID)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 175779-175792
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2993752460
- Cited by
- 37
- References
- 26
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 5 · 2023: 7 · 2022: 13 · 2021: 4 · 2020: 4
- FWCI
- 2.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Hibah PTUPT 2019, Ministry of Research, Technology, and Higher Education, Indonesia
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Engineering › Media Technology › Smart Cities and Technologies
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Customer Service Quality and Loyalty
- Keywords
- quality of experience, Internet of Things services, mean opinion score, user satisfaction evaluation, smart cities, public services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
