Expert evaluation methods

Abstract

An expert is a person who has made all the mistakes that can be made in a very narrow field. (Niels Bohr) Introduction Comprehensive, generalizable evaluations of digital libraries (DLs) are rare. Where evaluation does occur, it is generally minimal. Saracevic (2004) analysed around 80 evaluations to conclude that, both in scientific research and in practice, thorough evaluations of DLs are rather the exception than the rule. The complexity of DL systems can be identified as one reason for this. Examining them in their entirety is not straightforward. Even when attempting to do so, we lack scientifically accepted concepts, approaches and models. Another reason is the allocated funding in DL projects. Evaluation is always a ‘must have’ stated by the funder, but is rarely supported by adequate resources. These reasons are, unfortunately, still valid at the time of writing. In this chapter the method of expert evaluation is presented and shown to be one possible way of addressing such problems. Expert evaluations are heuristic or qualitative in nature, as opposed to quantitative evaluations, which aim to provide statistically significant results. Heuristic evaluations (Nielsen, 1994) are common in usability engineering, where user interfaces are evaluated by a small group of experts on the basis of their conformity to certain usability principles (heuristics). Expert evaluations differ from other types of heuristic evaluation in that they lack predefined heuristics. The experts…