Improve Reputation Evaluation of Crowdsourcing Participants Using Multidimensional Index and Machine Learning Techniques
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Building a scientific and reasonable reputation evaluation mechanism for crowdsourcing participants is an effective way to solve the problem of transaction fraud, to establish the trust of traders and ensure the quality of task completion. Under the big data environment, machine learning methods have been applied in the domain of e-commerce of physical goods to improve the traditional reputation evaluation methods, and achieved good results. However, few studies have applied machine learning methods to crowdsourcing, a form of service e-commerce, to evaluate the reputation of participants. This paper proposes a reputation evaluation model (i.e. LDA-RF) for crowdsourcing participants of Random Forest based on Linear Discriminant Analysis. The model consists of five steps: firstly, building a multidimensional reputation evaluation index system for crowdsourcing participants, collecting real data sets, and preprocessing data; secondly, data dimensionality reduction methods, including Linear Discriminant Analysis, Principal Component Analysis, Mean Impact Value method and ReliefF feature selection method, are used to eliminate redundant variables; thirdly, data normalization; fourthly, with selected feature subset, five machine learning techniques, Random Forest, Decision Tree, Back propagation Neural Network, Radial Basis Function Neural Network and Support Vector Machine are used to train the model; Fifthly, the validity of the model is tested by four evaluation measures: 10 fold cross validation, confusion matrix, Kruskal-wallis test and dispersion degree. The results show that the LDA-RF model on accuracy, F1-measure, generalization ability and robustness are better than those of other models, and it has better performance and effectiveness. This study represents a new contribution to establish reputation evaluation of crowdsourcing participants under big data environment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Highly accurate results can be ensured through the use of ML for trust evaluation in instances in which big data are processed.”
From Security and Trust Management in the Internet of Vehicles (IoV): Challenges and Machine Learning Solutions (2024) · cites this paper as [40]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 3×
“Set α1 ≥ . . . ≥ αd ≥ 0 and β1, . . . , βd as generalized eigenvalues and generalized eigenvectors of S(b), S(w) respectively: S(b)β = αS(w)β (6) Find the solution of LDA: T̂ = (β1, . . . , βm)T (7) C. RANDOM FOREST Random Forest (RF) is an ensemble machine learning algorithm based on decision tree classifier proposed…”
MethodInfluentialCortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445 · cited 1×
“SVM transforms data into high-dimensional space through kernel function, and realizes data linearly separable in high dimensional space [31].”
MethodA review of feature selection techniques in bioinformatics
Saeys, Y., Inza, I., & Larrañaga, P. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm344 · Cited by 5,244
Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks
Broomhead, D. S., & Lowe, D. (1988) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 3,443
A survey of trust and reputation systems for online service provision
Jøsang, A., Ismail, R., & Boyd, C. (2007) · Decision Support Systems · doi:10.1016/j.dss.2005.05.019 · Cited by 3,330 · cited 1×
“The basic idea of crowdsourcing participants’ reputation evaluation mechanism is to let the two parties involved in the transaction evaluate the other party after the transaction is completed [7].”
BackgroundA Practical Approach to Feature Selection
Kira, K., & Rendell, L. A. (1992) · Machine Learning Proceedings 1992 · doi:10.1016/b978-1-55860-247-2.50037-1 · Cited by 2,978 · cited 1×
“[27] proposed Relief algorithm, which is suitable for classification of two types of data.”
MethodEstimating attributes: Analysis and extensions of RELIEF
Kononenko, I. (1994) · Lecture Notes in Computer Science · doi:10.1007/3-540-57868-4_57 · Cited by 2,958 · cited 1×
“Kononenko [28] extended it in 1994, and got ReliefF algorithm to deal with multi-class problems.”
(2008) · Cited by 1,657
Neural network credit scoring models
West, D. (2000) · Computers & Operations Research · doi:10.1016/s0305-0548(99)00149-5 · Cited by 1,033 · cited 1×
“BPNN usually has three or more layers, including input layer, middle layer (hidden layer) and output layer [29].”
BackgroundCrowdsourcing: Why the Power of the Crowd Is Driving the Future of Business
(2009) · Journal of Consumer Marketing · doi:10.1108/07363760910965918 · Cited by 882 · cited 1×
“Procter & Gamble integrates the wisdom of tens of thousands of technical experts around the world and seeks solutions through crowdsourcing platform, which improves its Research and Development capability by 60%, increases the proportion of external innovation by 35%, and reduces Research and Development cost by 20% […”
BackgroundFrom Conservation to Crowdsourcing: A Typology of Citizen Science
Wiggins, A., & Crowston, K. (2011) · 2011 44th Hawaii International Conference on System Sciences · doi:10.1109/hicss.2011.207 · Cited by 700 · cited 1×
“Although crowdsourcing activities can bring together experts in relevant domains to provide valuable results [5], [6], there are still quite a few task publishers who are concerned about the potential risks of crowdsourcing activities, being the non-physical commodities traded in crowdsourcing are facing the problems…”
BackgroundTrust management through reputation mechanisms
Zacharia, G., & Maes, P. (2000) · Applied Artificial Intelligence · doi:10.1080/08839510050144868 · Cited by 521 · cited 1×
“RESEARCH MOTIVATION AND OBJECTIVES Reputation mechanism is developed in the context of e-commerce [8], which has been widely used in practice.”
BackgroundEnsemble boosted trees with synthetic features generation in application to bankruptcy prediction
Zięba, M., Tomczak, S. K., & Tomczak, J. M. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.04.001 · Cited by 418 · cited 1×
“The ensemble model has been studied by scholars in the field of credit evaluation for bank users and achieved good prediction results [32], [33].”
BackgroundData quality from crowdsourcing: a study of annotation selection criteria
Hsueh, P.-Y., Melville, P., & Sindhwani, V. (2009) · Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing - HLT '09 · doi:10.3115/1564131.1564137 · Cited by 251 · cited 1×
“Although crowdsourcing activities can bring together experts in relevant domains to provide valuable results [5], [6], there are still quite a few task publishers who are concerned about the potential risks of crowdsourcing activities, being the non-physical commodities traded in crowdsourcing are facing the problems…”
BackgroundA comparative study on base classifiers in ensemble methods for credit scoring
Abellán, J., & Castellano, J. G. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.12.020 · Cited by 248
SACRM: Social Aware Crowdsourcing with Reputation Management in mobile sensing
Ren, J., Zhang, Y., Zhang, K., & Shen, X. (2015) · Computer Communications · doi:10.1016/j.comcom.2015.01.022 · Cited by 97
Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms
Gaikwad, S. S., Morina, D., Ginzberg, A., Mullings, C. A., Goyal, S., Gamage, D., Diemert, C., Burton, M., Zhou, S., Whiting, M. E., et al. (2016) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2984511.2984542 · Cited by 56 · cited 1×
“[18] designed boomerang reputation evaluation system based on game theory, to incentive task publishers reporting private information more accurately, and to make fair and true evaluation of crowdsourcing participants.”
BackgroundIncentive Mechanisms for Crowdsourcing Platforms
Katmada, A., Σάτσιου, Ά., & Kompatsiaris, I. Y. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45982-0_1 · Cited by 46 · cited 1×
“[17] proposed to offer implicit and explicit mixed rewards to crowdsourcing participants, in order to attract participants with different motivations to participate in crowdsourcing actively and submit high-quality results.”
BackgroundRTRC: a reputation-based incentive game model for trustworthy crowdsourcing service
Ma, X., Ma, J., Li, H., Jiang, Q. G., & Gao, S. (2016) · China Communications · doi:10.1109/cc.2016.7897544 · Cited by 21 · cited 1×
“[15] proposed game-based incentive mechanism RTRC to encourage the crowd to provide authentic and credible information in mobile crowdsourcing.”
BackgroundDesign and Analysis of Incentive and Reputation Mechanisms for Online Crowdsourcing Systems
Xie, H. L., Lui, J. C. S., & Towsley, D. (2016) · ACM Transactions on Modeling and Performance Evaluation of Computing Systems · doi:10.1145/2897510 · Cited by 18 · cited 1×
“[16] designed the reputation model of crowdsourcing participants to improve the tasks quality by balancing system efficiency and transaction cost.”
BackgroundQnQ: A reputation model to secure mobile crowdsourcing applications from incentive losses
Bhattacharjee, S., Ghosh, N., Shah, V. K., & Das, S. K. (2017) · IEEE Conference on Communications and Network Security · doi:10.1109/cns.2017.8228635 · Cited by 17 · cited 1×
“[20] put forwarded QnQ reputation model, and used regression method to identify honest, selfish and fraudulent crowdsourcing participants.”
MethodStudying in the ‘Bazaar’: An Exploratory Study of Crowdsourced Learning in GitHub
Lu, Y., Mao, X., Wang, T., Yin, G., Li, Z., & Wang, W. (2019) · IEEE Access · doi:10.1109/access.2019.2915247 · Cited by 13 · cited 1×
“Crowdsourcing provides a new way for crowd to participate in value creation full-time or part-time and solve problems in a long distance [2].”
BackgroundTask-Management Method Using R-Tree Spatial Cloaking for Large-Scale Crowdsourcing
Li, Y., & Shin, B. (2017) · Symmetry · doi:10.3390/sym9120311 · Cited by 5 · cited 1×
“Crowdsourcing through the Internet calls for idle human resources to participate in value creation [4], changes the traditional mode of value creation, achieves the multiplier effect of gathering people’s wisdom, realizes the effective integration of social resources, and improves the efficiency of social operation.”
BackgroundResearch on Witkey Credit Ability Evaluation Mechanism
Zhao, W. (2012) · Cited by 1
Research works printed but not linked: 11
- broomhead, 1988, Multivariable functional interpolation and adaptive networks, Complex Syst, 2, 321
- liao, 2014, Research on C2C seller selection model based on the evaluation of implicit credit factors, Journal of Yunnan University of Finance and Economics, 30, 134
- wang, 2014, B2C enterprise credit evaluation based on improved TOPSIS method, Soft Science, 28, 21
- fu, 2016, Network supplier reputation evaluation model based on big data, Journal of Central University of Finance & Economics, 8, 74
- rong, 2018, The study of credit evaluation of suppliers on cross-border E-commerce platform based on big data, Statist Inf Forum, 33, 100
- yan, 2017, Reputation model of crowdsourcing workers based on activity degree, J Comput Appl, 37, 2039
- li, 2012, Multidimensional reputation evaluation model for C2C E-commerce, Chinese Journal of Management, 9, 204
- howe, 2006, The rise of crowdsourcing, Wired Mag, 14, 1
- ruan, 2015, Crowdsourcing quality control based on reputation model of Dempster–Shafer theory, J Comput Appl, 35, 2380
- wang, 2016, Online incentive mechanism based on reputation for mobile crowdsourcing system, J Comput Appl, 36, 2121
- rui, 2016, Reputation-based incentive mechanism in crowdsourcing, J Electron Inf Technol, 38, 1808
Cited by
A survey on trust evaluation based on machine learning
Wang, J., Jing, X., Yan, Z., Fu, Y., Pedrycz, W., & Yang, L. T. (2021) · ACM Computing Surveys · doi:10.1145/3408292 · Cited by 136
“Imana et al. [2010] offered a method, which uses Radial Basis Function Neural Networks (RBS-NN), to predict the trust of the nodes in Mobile Ad Hoc Networks (MANETs).”
BackgroundMethodAlalwany, E., & Mahgoub, I. (2024) · Sensors · doi:10.3390/s24020368 · Cited by 130
“The high accuracy in trust evaluation when one uses big data is due to the adequate data sources big data provide [40].”
BackgroundIoT trust and reputation: a survey and taxonomy
Aaqib, M., Ali, A., Chen, L., & Nibouche, O. (2023) · Journal of Cloud Computing Advances Systems and Applications · doi:10.1186/s13677-023-00416-8 · Cited by 93
“Computational overhead is rare in some approaches like [115, 125, 127, 130] and does not pay special attention to evaluating the trust efficiency.”
BackgroundSentiment Classification of Crowdsourcing Participants’ Reviews Text Based on LDA Topic Model
Huang, Y., Wang, R., Huang, B., Wei, B., Zheng, S. L., & Chen, M. (2021) · IEEE Access · doi:10.1109/access.2021.3101565 · Cited by 56
“The crowdsourcing platform has rich reviews, and is also an important platform for many scholars to research [11], [25], [26], which is applicable to the research of this study.”
BackgroundSelf-citationZhang, Z., Jing, J., Wang, X., Choo, K. R., & Gupta, B. B. (2020) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-020-00230-0 · Cited by 29
Metadata
- Authors
- Huang, Y., & Chen, M.
- Affiliations
- Wuhan University (CN)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 118055-118067
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2964583479
- Cited by
- 25
- References
- 24
- Citations by year
- 2025: 4 · 2024: 2 · 2023: 5 · 2022: 7 · 2021: 2 · 2020: 5
- FWCI
- 3.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (91118003, 61373039)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Networks and Communications › Access Control and Trust
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Evaluation and Optimization Models
- Keywords
- reputation measurement, crowdsourcing, transaction fraud, trust, linear discriminant analysis, random forest, feature selection, dimensionality reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.