A Field Evaluation of Natural Language for Data Retrieval

Although a large number of natural language database interfaces have been developed, there have been few empirical studies of their practical usefulness. This paper presents the design and results of a field evaluation of a natural language system-NLS-used for data retrieval. A balanced multifactorial design comparing NLS to a reference retrieval language SQL is described. The data are analyzed on two levels: Work task (n=87) and query (n=1081). SQL performed better than NLS on a variety of measures, but NLS required less effort to use. Subjects performed much poorer than expected based on the results of laboratory studies. This finding is attributed to the complexity of the field setting and to optimism in grading laboratory experiments. The methodology developed for studying computer languages in real work settings was successful in consistently measuring differences in treatments over a variety of conditions.