TY - GEN
T1 - Study on patterns and effect of task diversity in software crowdsourcing
AU - Mejorado, Denisse Martinez
AU - Saremi, Razieh
AU - Yang, Ye
AU - Ramirez-Marquez, Jose E.
N1 - Publisher Copyright:
© 2020 IEEE Computer Society. All rights reserved.
PY - 2020/10/5
Y1 - 2020/10/5
N2 - Context: The success of software crowdsourcing depends on steady pools of task demand and active workers supply. Existing analysis reveals an average task failure ratio of 15.7% in software crowdsourcing market. Goal: The objective of this study is to empirically investigate patterns and effect of task diversity in software crowdsourcing platform in order to improve the success and efficiency of software crowdsourcing. Method: We first propose a conceptual task diversity model, and develop an approach to measuring and analyzing task diversity. More specifically, task diversity is characterized based on semantic similarity, dynamic competition level, and the analysis includes identifying the dominant attributes distinguishing the competition levels, and measuring the impact of task diversity on task success and worker performance in crowdsourcing platform. The empirical study is conducted on more than one year's real-world data from TopCoder, one of the leading software crowdsourcing platforms. Results: We identified that monetary prize and task complexity are the dominant attributes that differentiate among different competition levels. Based on these dominant attributes, we concluded three task diversity patterns (configurations) from workers behavior perspective: responsive-to-prize, responsive-to-prize-andcomplexity and over-responsive-to-prize. This study supports that the second pattern, i.e. responsive-to-prize-and-complexity configuration, associates with the lowest task failure ratio. Conclusions: These findings are helpful for task requesters to plan for and improve task success in a more effective and efficient manner in software crowdsourcing platform.
AB - Context: The success of software crowdsourcing depends on steady pools of task demand and active workers supply. Existing analysis reveals an average task failure ratio of 15.7% in software crowdsourcing market. Goal: The objective of this study is to empirically investigate patterns and effect of task diversity in software crowdsourcing platform in order to improve the success and efficiency of software crowdsourcing. Method: We first propose a conceptual task diversity model, and develop an approach to measuring and analyzing task diversity. More specifically, task diversity is characterized based on semantic similarity, dynamic competition level, and the analysis includes identifying the dominant attributes distinguishing the competition levels, and measuring the impact of task diversity on task success and worker performance in crowdsourcing platform. The empirical study is conducted on more than one year's real-world data from TopCoder, one of the leading software crowdsourcing platforms. Results: We identified that monetary prize and task complexity are the dominant attributes that differentiate among different competition levels. Based on these dominant attributes, we concluded three task diversity patterns (configurations) from workers behavior perspective: responsive-to-prize, responsive-to-prize-andcomplexity and over-responsive-to-prize. This study supports that the second pattern, i.e. responsive-to-prize-and-complexity configuration, associates with the lowest task failure ratio. Conclusions: These findings are helpful for task requesters to plan for and improve task success in a more effective and efficient manner in software crowdsourcing platform.
KW - Software crowdsourcing
KW - Task diversity
KW - Task failure
KW - Task success
KW - Worker performance
UR - http://www.scopus.com/inward/record.url?scp=85095837577&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85095837577&partnerID=8YFLogxK
U2 - 10.1145/3382494.3410689
DO - 10.1145/3382494.3410689
M3 - Conference contribution
AN - SCOPUS:85095837577
T3 - International Symposium on Empirical Software Engineering and Measurement
BT - ESEM 2020 - Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
T2 - 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2020
Y2 - 5 October 2020 through 7 October 2020
ER -