TY - JOUR
T1 - A Field Evaluation of Natural Language for Data Retrieval
AU - Jarke, Matthias
AU - Turner, Jon A.
AU - Stohr, Edward A.
AU - Vassiliou, Yannis
AU - White, Norman H.
AU - Michielsen, Ken
PY - 1985/1
Y1 - 1985/1
N2 - Although a large number of natural language database interfaces have been developed, there have been few empirical studies of their practical usefulness. This paper presents the design and results of a field evaluation of a natural language system-NLS-used for data retrieval. A balanced multifactorial design comparing NLS to a reference retrieval language SQL is described. The data are analyzed on two levels: Work task (n=87) and query (n=1081). SQL performed better than NLS on a variety of measures, but NLS required less effort to use. Subjects performed much poorer than expected based on the results of laboratory studies. This finding is attributed to the complexity of the field setting and to optimism in grading laboratory experiments. The methodology developed for studying computer languages in real work settings was successful in consistently measuring differences in treatments over a variety of conditions.
AB - Although a large number of natural language database interfaces have been developed, there have been few empirical studies of their practical usefulness. This paper presents the design and results of a field evaluation of a natural language system-NLS-used for data retrieval. A balanced multifactorial design comparing NLS to a reference retrieval language SQL is described. The data are analyzed on two levels: Work task (n=87) and query (n=1081). SQL performed better than NLS on a variety of measures, but NLS required less effort to use. Subjects performed much poorer than expected based on the results of laboratory studies. This finding is attributed to the complexity of the field setting and to optimism in grading laboratory experiments. The methodology developed for studying computer languages in real work settings was successful in consistently measuring differences in treatments over a variety of conditions.
KW - Human-machine interaction
KW - interface design
KW - language evaluation
KW - natural language query
KW - query languages
UR - http://www.scopus.com/inward/record.url?scp=0021785397&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=0021785397&partnerID=8YFLogxK
U2 - 10.1109/TSE.1985.231847
DO - 10.1109/TSE.1985.231847
M3 - Article
AN - SCOPUS:0021785397
SN - 0098-5589
VL - SE-11
SP - 97
EP - 114
JO - IEEE Transactions on Software Engineering
JF - IEEE Transactions on Software Engineering
IS - 1
ER -