Assessing Data Quality: From Concordance, through Correctness and Completeness, to Valid Manipulatable Representations

Abstract

The papers by Stein et al.1 and Aronsky and Haug2 address the quality of the data found in clinical record systems. Stein et al. approach the problem as one of internal consistency. Their paper explores concordance within record systems, exploring the extent to which evidence found in one part of a clinical database is consistent with evidence found in another part. Specifically, they examine agreement between entries in a free-text narrative field with data found in coded fields. Aronsky and Haug examine concordance across two different clinical record systems—the HELP computerized clinical record system and a reference standard consisting of the sum of all information available in the paper chart and the computerized clinical record. Aronsky and Haug complement their appraisal of concordance with an outcome evaluation, determining the level of agreement in clinical severity indexes resulting from the information contained in the different record systems. Stein et al. use the measure of internal concordance to alert users of a computerized record system to the fact that they may get misleading answers unless they query each field that might contain a piece of information and resolve any discrepancies. Aronsky and Haug argue the need for equivalence in recommendations based on the clinical record, regardless of which form of a clinical record is used; they are saying, in effect, that records with higher concordance should lead to similar recommendations. The use of the term “concordance” is appropriate for framing the question posed in each of these papers. Concordance originated in the 14th century church, referring to a companion text to an original document, the companion text consisting of an enumeration of all terms appearing in the original document. These authors extended the target object from “terms” to “concepts” and restricted the enumeration of concepts to only those relevant to specific clinical phenomena. The authors' uses of the term “concordance” implies that the question of interest is the extent to which all clinically significant concepts found in one section (Stein et al.) or form (Aronsky and Haug) are evident in another section or form. The evidence presented in the two papers clearly indicates that the two sections or forms are not in complete accord, and the discussions evaluate the consequences of the discord. Clinical records, be they paper or electronic, are no more and no less than representations of the true state of the patient and the events occurring during the process of care for the patient. Concordance is a characteristic of representations: It is possible to estimate the degree of similarity within or between representations, such as patient records. While one could dispute the merits of different computational forms employed to characterize concordance, the basic intent of such a statistic is valid—to gauge the level of agreement between two documents. At the same time, it is critical to recognize that concordance offers no indication as to whether the representations themselves are true and accurate depictions of the real state of the patient. Clinical data are a scarce and expensive resource. These studies advance our understanding of the degree to which we can re-use data recorded in today's clinical information systems for practice management, decision support, and clinical or health services research. This type of work should be extended in two directions—prospective studies of data accuracy in extant clinical records and methodological studies of strategies to produce more robust language structures for representing clinical phenomena. Hogan and Wagner3 provide a model for examining data accuracy by assessing correctness and completeness. The approach they use enhances the concordance studies not only by examining data in the clinical record but by prospectively constructing a gold standard so that the patient and care provider can be used as information sources. In essence, this approach goes further than concordance to ensure that the record is a correct representation of the state of the patient. At best, however, data concordance and accuracy studies are only as good as the underlying representations. These studies have an inherent limitation, because they rely on the vocabulary primitive of term or phrase. Almost all studies in this realm employ some type of parsing strategy to select specific words or phrases on which to evaluate agreement. These words and phrases themselves may be overly limiting the ability of clinical records to represent the true state of the patient, because they force the reduction of expressions of complex clinical phenomena into atomic words and phrases. Studies based on the vocabulary primitive of “terms” rather than on more sophisticated representations of patient phenomena remain restricted by the fundamental nature of records. It is possible to envision language structures in the clinical record that are more robust than simple wo