TY - GEN
T1 - Evaluating design solutions using crowds
AU - Bao, Jin
AU - Sakamoto, Yasuaki
AU - Nickerson, Jeffrey V.
PY - 2011
Y1 - 2011
N2 - Crowds can be used to generate and evaluate design solutions. To increase a crowdsourcing system's effectiveness, we propose and compare two evaluation methods, one using five-point Liked scale rating and the other premction voting. Our results indicate that although the two evaluation methods correlate, they have mfferent goals: whereas prediction voting focuses evaluators on identifying the very best solutions, the rating focuses evaluators on the entire range of solutions. Thus, premction voting is appropriate when there are many poor quality solutions that need to be filtered out, and rating is suited when all ideas are reasonable and mstinctions need to be made across all solutions. The crowd prefers participating in premction voting. The results have pragmatic implications, suggesting that evaluation methods should be assigned in relation to the mstnbution of quality present at each stage of crowdsourcing.
AB - Crowds can be used to generate and evaluate design solutions. To increase a crowdsourcing system's effectiveness, we propose and compare two evaluation methods, one using five-point Liked scale rating and the other premction voting. Our results indicate that although the two evaluation methods correlate, they have mfferent goals: whereas prediction voting focuses evaluators on identifying the very best solutions, the rating focuses evaluators on the entire range of solutions. Thus, premction voting is appropriate when there are many poor quality solutions that need to be filtered out, and rating is suited when all ideas are reasonable and mstinctions need to be made across all solutions. The crowd prefers participating in premction voting. The results have pragmatic implications, suggesting that evaluation methods should be assigned in relation to the mstnbution of quality present at each stage of crowdsourcing.
KW - Creativity
KW - Crowdsourcing
KW - Evaluation
KW - Human-based genetic algorithms
KW - Mechanical turk
UR - http://www.scopus.com/inward/record.url?scp=84870319587&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84870319587&partnerID=8YFLogxK
U2 - 10.2139/ssrn.2201651
DO - 10.2139/ssrn.2201651
M3 - Conference contribution
AN - SCOPUS:84870319587
SN - 9781618390981
T3 - 17th Americas Conference on Information Systems 2011, AMCIS 2011
SP - 3923
EP - 3931
BT - 17th Americas Conference on Information Systems 2011, AMCIS 2011
T2 - 17th Americas Conference on Information Systems 2011, AMCIS 2011
Y2 - 4 August 2011 through 8 August 2011
ER -