Improve Reputation Evaluation of Crowdsourcing Participants Using Multidimensional Index and Machine Learning Techniques
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Building a scientific and reasonable reputation evaluation mechanism for crowdsourcing participants is an effective way to solve the problem of transaction fraud, to establish the trust of traders and ensure the quality of task completion. Under the big data environment, machine learning methods have been applied in the domain of e-commerce of physical goods to improve the traditional reputation evaluation methods, and achieved good results. However, few studies have applied machine learning methods to crowdsourcing, a form of service e-commerce, to evaluate the reputation of participants. This paper proposes a reputation evaluation model (i.e. LDA-RF) for crowdsourcing participants of Random Forest based on Linear Discriminant Analysis. The model consists of five steps: firstly, building a multidimensional reputation evaluation index system for crowdsourcing participants, collecting real data sets, and preprocessing data; secondly, data dimensionality reduction methods, including Linear Discriminant Analysis, Principal Component Analysis, Mean Impact Value method and ReliefF feature selection method, are used to eliminate redundant variables; thirdly, data normalization; fourthly, with selected feature subset, five machine learning techniques, Random Forest, Decision Tree, Back propagation Neural Network, Radial Basis Function Neural Network and Support Vector Machine are used to train the model; Fifthly, the validity of the model is tested by four evaluation measures: 10 fold cross validation, confusion matrix, Kruskal-wallis test and dispersion degree. The results show that the LDA-RF model on accuracy, F1-measure, generalization ability and robustness are better than those of other models, and it has better performance and effectiveness. This study represents a new contribution to establish reputation evaluation of crowdsourcing participants under big data environment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Highly accurate results can be ensured through the use of ML for trust evaluation in instances in which big data are processed.”
From Security and Trust Management in the Internet of Vehicles (IoV): Challenges and Machine Learning Solutions (2024) · cites this paper as [40]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 3×
“Set α1 ≥ . . . ≥ αd ≥ 0 and β1, . . . , βd as generalized eigenvalues and generalized eigenvectors of S(b), S(w) respectively: S(b)β = αS(w)β (6) Find the solution of LDA: T̂ = (β1, . . . , βm)T (7) C. RANDOM FOREST Random Forest (RF) is an ensemble machine learning algorithm based on decision tree classifier proposed…”
MethodInfluentialCortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445 · cited 1×
“SVM transforms data into high-dimensional space through kernel function, and realizes data linearly separable in high dimensional space [31].”
MethodA review of feature selection techniques in bioinformatics
Saeys, Y., Inza, I., & Larrañaga, P. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm344 · Cited by 5,244
Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks
Broomhead, D. S., & Lowe, D. (1988) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 3,443
A survey of trust and reputation systems for online service provision
Jøsang, A., Ismail, R., & Boyd, C. (2007) · Decision Support Systems · doi:10.1016/j.dss.2005.05.019 · Cited by 3,330 · cited 1×
“The basic idea of crowdsourcing participants’ reputation evaluation mechanism is to let the two parties involved in the transaction evaluate the other party after the transaction is completed [7].”
BackgroundA Practical Approach to Feature Selection
Kira, K., & Rendell, L. A. (1992) · Machine Learning Proceedings 1992 · doi:10.1016/b978-1-55860-247-2.50037-1 · Cited by 2,978 · cited 1×
“[27] proposed Relief algorithm, which is suitable for classification of two types of data.”
MethodEstimating attributes: Analysis and extensions of RELIEF
Kononenko, I. (1994) · Lecture Notes in Computer Science · doi:10.1007/3-540-57868-4_57 · Cited by 2,958 · cited 1×
“Kononenko [28] extended it in 1994, and got ReliefF algorithm to deal with multi-class problems.”
(2008) · Cited by 1,657
Neural network credit scoring models
West, D. (2000) · Computers & Operations Research · doi:10.1016/s0305-0548(99)00149-5 · Cited by 1,033 · cited 1×
“BPNN usually has three or more layers, including input layer, middle layer (hidden layer) and output layer [29].”
Background
Research works printed but not linked: 11
- broomhead, 1988, Multivariable functional interpolation and adaptive networks, Complex Syst, 2, 321
- liao, 2014, Research on C2C seller selection model based on the evaluation of implicit credit factors, Journal of Yunnan University of Finance and Economics, 30, 134
- wang, 2014, B2C enterprise credit evaluation based on improved TOPSIS method, Soft Science, 28, 21
- fu, 2016, Network supplier reputation evaluation model based on big data, Journal of Central University of Finance & Economics, 8, 74
- rong, 2018, The study of credit evaluation of suppliers on cross-border E-commerce platform based on big data, Statist Inf Forum, 33, 100
- yan, 2017, Reputation model of crowdsourcing workers based on activity degree, J Comput Appl, 37, 2039
- li, 2012, Multidimensional reputation evaluation model for C2C E-commerce, Chinese Journal of Management, 9, 204
- howe, 2006, The rise of crowdsourcing, Wired Mag, 14, 1
- ruan, 2015, Crowdsourcing quality control based on reputation model of Dempster–Shafer theory, J Comput Appl, 35, 2380
- wang, 2016, Online incentive mechanism based on reputation for mobile crowdsourcing system, J Comput Appl, 36, 2121
- rui, 2016, Reputation-based incentive mechanism in crowdsourcing, J Electron Inf Technol, 38, 1808
Cited by
A survey on trust evaluation based on machine learning
Wang, J., Jing, X., Yan, Z., Fu, Y., Pedrycz, W., & Yang, L. T. (2021) · ACM Computing Surveys · doi:10.1145/3408292 · Cited by 136
“Imana et al. [2010] offered a method, which uses Radial Basis Function Neural Networks (RBS-NN), to predict the trust of the nodes in Mobile Ad Hoc Networks (MANETs).”
BackgroundMethodAlalwany, E., & Mahgoub, I. (2024) · Sensors · doi:10.3390/s24020368 · Cited by 130
“The high accuracy in trust evaluation when one uses big data is due to the adequate data sources big data provide [40].”
BackgroundIoT trust and reputation: a survey and taxonomy
Aaqib, M., Ali, A., Chen, L., & Nibouche, O. (2023) · Journal of Cloud Computing Advances Systems and Applications · doi:10.1186/s13677-023-00416-8 · Cited by 93
“Computational overhead is rare in some approaches like [115, 125, 127, 130] and does not pay special attention to evaluating the trust efficiency.”
BackgroundSentiment Classification of Crowdsourcing Participants’ Reviews Text Based on LDA Topic Model
Huang, Y., Wang, R., Huang, B., Wei, B., Zheng, S. L., & Chen, M. (2021) · IEEE Access · doi:10.1109/access.2021.3101565 · Cited by 56
“The crowdsourcing platform has rich reviews, and is also an important platform for many scholars to research [11], [25], [26], which is applicable to the research of this study.”
BackgroundSelf-citationZhang, Z., Jing, J., Wang, X., Choo, K. R., & Gupta, B. B. (2020) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-020-00230-0 · Cited by 29
Fu, Z.-L., Cong, J., Wang, L., Shi, L., & Guo, W. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101871 · Cited by 24
Tripartite evolutionary game analysis of the collective intelligence design ecosystem
Fu, Z.-L., Wang, L., Xue, S.-J., Ma, J., Zhang, J., & Guo, W. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.135217 · Cited by 18
Forecast of Fossil Fuel Demand Based On Low Carbon Emissions from the Perspective of Energy Security
Huang, Y., Lin, J., Wang, Y., Min, J., Wang, R., Jin, D., & Wei, B. (2023) · Chemistry and Technology of Fuels and Oils · doi:10.1007/s10553-023-01490-z · Cited by 12
“The integrated model has a higher prediction accuracy than the single model [23, 24].”
MethodSelf-citationReputation aware optimal team formation for collaborative software crowdsourcing in industry 5.0
Akter, S. N., Sinthia, A. K., Roy, P., Razzaque, M. A., Hassan, M. M., Pupo, F., & Fortino, G. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101710 · Cited by 11
“Additionally, there needs to be an efficient reputation mechanism that matches both the institutional and field-specific experience of the workers (Huang and Chen, 2019).”
InfluentialWITHDRAWN: E-commerce brand marketing based on FPGA and machine learning
Wang, Q., Cai, R., & Zhao, M. (2020) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2020.103446 · Cited by 11
RetractedEcological network evolution analysis in collective intelligence design ecosystem
Fu, Z.-L., Guo, W., Wang, L., Ma, J., Shi, L., & Lin, M. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102150 · Cited by 10
Quantitative evaluation of crowd intelligence innovation system health: An ecosystem perspective
Zheng, Q. S., Guo, W., Ding, G., Zhang, H., Fu, Z.-L., Qin, S., & Peng, W. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102423 · Cited by 7
Trust Evaluation Method of E-Commerce Enterprises with High-Involvement Experience Products
Liang, K., Jun, H., & Wu, P. (2022) · Sustainability · doi:10.3390/su142315562 · Cited by 4
Fang, H. (2022) · Wireless Communications and Mobile Computing · doi:10.1155/2022/7395202 · Cited by 4
“It is in this situation that machine learning theory gradually becomes one of the core elements of AI research [7].”
BackgroundRetractedSivachandiran, S. (2020) · International Journal of Advanced Trends in Computer Science and Engineering · doi:10.30534/ijatcse/2020/106942020 · Cited by 2
A Crowdsourcing-based Localization Scheme with Ultra-Wideband Communication
Shi, B., Hu, Y., Duan, Q., Han, L., & Ding, Y. (2020) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon43393.2020.9255352 · Cited by 2
“With an open framework, crowdsourcing enables decentralized devices/individuals/groups to collaborate efficiently [1] for a common purpose, such as information retrieval, natural language processing, and machine learning [2].”
BackgroundReputation Aware Fair Worker Selection in Collaborative Software Crowdsourcing
Rahman, S., Sinthia, A. K., Akter, S. N., Roy, P., & Razzaque, M. A. (2021) · International Conference on Sustainable Technologies for Industry 5.0 (STI) · doi:10.1109/sti53101.2021.9732415 · Cited by 1
Babu, B. N., & Gunasekaran, M. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-981-96-4410-0_27 · Cited by 1
Movahedkor, N., Bagheri, A., & Mirtaheri, S. L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4130134
Huang, Y., Zheng, Z., & Wei, B. (2022) · Mobile Information Systems · doi:10.1155/2022/5008465
“Based on the reputation evaluation index system of crowdsourcing participants established in [2], the evaluation indexes are collected.”
BackgroundSelf-citationYang, Q., Cai, L., Gong, J., Tang, M., & Lin, Y. (2022) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c57518.2022.00040
“Huang et al. [12] designed a stochastic forest crowd-sourcing participant reputation evaluation model based on linear discriminant analysis in view of the fact that few studies applied machine learning methods to crowd-sourcing to evaluate the credibility of participants.”
MethodHuang, Y., & Yu, Q. (2022) · International Conference on Computer Science and Management Technology (ICCSMT) · doi:10.1109/iccsmt58129.2022.00056
“The influence of crowdsourcing participants' credit rating on transaction performance At present, the reputation evaluation of platform crowdsourcing participants mainly adopts two indicators, transaction amount and transaction evaluation [1, 2].”
BackgroundSelf-citationHuang, Y., Zhao, P., Qin, Y., Musa, M. A., & Wang, Y. (2025) · International Conference on Cyber Security, Artificial Intelligence and Digital Economy (CSAIDE) · doi:10.1145/3729706.3729795
“Huang et al. [ 6 ] built a reputation evaluation index system for crowdsourcing service providers and used a machine learning method to evaluate the reputation.”
Self-citationA Review on Efficient Trust Computational and Evaluation Based on Machine Learning
Kumar, A., Bhatia, A., & Kumar, D. S. (2025) · International Journal of Inventive Engineering and Sciences · doi:10.35940/ijies.k1136.12111125
Trustworthy AI in Spatial Crowdsourcing: Mitigating Bias and Detecting Outliers for Robust Models
Rahman, M. M., Hasan, M. M., Mamun, Q., Wu, J., Pan, Z., & Shimamoto, S. (2025) · Asia-Pacific Conference on Communications (APCC) · doi:10.23919/apcc64555.2025.11279827
“To establish thresholds for consistency, we use robust z-scores [14], assuming stable behaviour unless the contributor is acting adversarially.”
Metadata
- Authors
- Huang, Y., & Chen, M.
- Affiliations
- Wuhan University (CN)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 118055-118067
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2964583479
- Cited by
- 25
- References
- 24
- Citations by year
- 2025: 4 · 2024: 2 · 2023: 5 · 2022: 7 · 2021: 2 · 2020: 5
- FWCI
- 3.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (91118003, 61373039)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Networks and Communications › Access Control and Trust
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Evaluation and Optimization Models
- Keywords
- reputation measurement, crowdsourcing, transaction fraud, trust, linear discriminant analysis, random forest, feature selection, dimensionality reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.