Partial-model testing as a validation tool for system dynamics (1983)

Abstract

System dynamicists have long emphasized the use of multiple data sources and multiple methods to estimate parameters and test models. Ideally, one should estimate parameters using data " below the level of aggregation of the model" . For example, in a model of capital investment, one could estimate the length and order of the construction delay directly from data on the construction times for a large sample of relevant projects. Often, however, the needed data are not available. At the other end of the spectrum one can use " whole model estimation" in which the parameters are found by fitting the behavior of the full model to the available aggregate time series data. Whole model estimation, however, often suffers from identification problems. In this 1983 paper Jack Homer describes partial model testing, in which parameters are estimated within a subset of model structure rather than by calibration of the entire model. Homer illustrates with an example from his research on the adoption of new medical innovations, specifically, the cardiac pacemaker. He illustrates the partial model testing process with an important formulation representing how the clinical and research communities carry out and report follow‐up data on the safety and efficacy of a medical innovation as it evolves. The paper also illustrates the necessity of careful empirical work to collect new data. Debates over methods to estimate parameters are sterile without the data to implement them. In the context of medical innovation, it would have been plausible to assume that follow‐up data evaluating pacemaker efficacy would grow smoothly over time as use expanded, though perhaps with a lag. Not content with such easy assumptions, however, Homer examined every article on pacing published in every issue of the relevant cardiology journals, from the creation of the pacemaker through the (then) present. The work, done years before the advent of the Internet and online databases, was painstaking and time consuming—the journals had to be searched and articles coded by hand. The payoff was a unique dataset documenting the actual dynamics of evaluative reporting. Rather than smooth growth, the data showed a pronounced oscillation in the publication of evaluative studies, even though other data, which Homer also assembled from original sources, showed smooth growth in adoption, clinical indications and pacemaker use. Homer's model generates the same oscillation endogenously, and the partial model tests provide robust estimates of the parameters governing the institutional processes (such as research and publication delays) and behavioral decision rules (such as the decision by researchers and clinicians to initiate new follow‐up studies) that determine the existence, period, and amplitude of the cycle. The structure identified through this process was not only important to accurately model the evolution of the pacemaker, but has important policy implications still relevant today. All modelers should follow Homer's example and put in the hard work to generate, from primary sources, the data needed to estimate the important parameters and relationships in our models. John Sterman Homer J. 1983. Partial‐model testing as a validation tool for system dynamics. In Proceedings of the 1983 International System Dynamics Conference . System Dynamics Society, Chestnut Hill, MA; 919–932. Abstract This paper discusses an approach to model refinement that involves testing the behavior of individual pieces of a model in response to empirical input data for comparison with empirical output data. Partial‐model tests should be used for selecting formulations or estimating parameters only when appropriate case‐specific or logical information is not available for this purpose. The smaller the model components used for partial‐model testing, the more likely it is that the model will prove useful for anticipating events outside historical experience and the less likely it is that observed behavior will be incorrectly attributed to certain relationships or parameters. Thus, from the standpoint of structural validity, partial‐model testing is an improvement over whole‐model testing for the purpose of structural adjustment. The paper presents a detailed example of partial‐model testing in the context of a generic model of the evolving use of a new medical technology. Specifically, the technique is used for adjusting and validating a model subsystem that can explain why the reporting of clinical information on cardiac pacemakers has been marked by regular oscillations over time. Copyright © 2012 System Dynamics Society.