Rate Distortion Theory: A Mathematical Basis for Data Compression

Abstract

Shannon's Information Theory created a great intellectual impact at the time of its publication. Aside from its intrinsic beauty, it presented the truly powerful concept of a definite channel capacity. This capacity, as must have been very surprising at the time, is measured in bits/second, rather than in units of bandwidth. "Rate Distortion Theory" played a very important role in the "Information Theory" formulated by Shannon: it provided a link between the traditional theory of modulation, and the material of discrete sources and channels. This topic, however, received almost no attention until the latter sixties. 1 An increasing amount of effort is now being devoted to this area, and Dr. Berger's book is therefore a very timely contribution. The fact that Rate Distortion Theory had been ignored for so long can only be explained by the strength of the impact carried in the other topics introduced by Shannon. The progression of information theoretic research has duplicated, in a surprisingly close way, the organization of Shannon's original papers. The initial effort was devoted to constructing source codes for discrete alphabets. A second wave covered codes and performance bounds for assorted analog and discrete channels. It is somewhat ironic that interest in such a "traditional" topic as the encoding of continuous sources revived at a time when many sources are, at last, truly digital. Dr. Berger's book is quite direct. After a short introductory chapter, it devotes three chapters to presenting what may be called the fundamentals of Rate Distortion Theory. The rate distortion function of a finite alphabet memoryless source is defined in Chapter 2, and the problem of calculating this function is carefully discussed. The significance of this function in coding and transmission is demonstrated in Chapter 3. Chapter 4 repeats the material of Chapter 2 for continuous-amplitude sources, as well as presentirig the important results on vector-valued and stationary Gaussian