Rate‐Distortion Theory

Abstract

Rate‐distortion theory is the branch of information theory that treats compressing the data produced by an information source down to a specified encoding rate that is strictly less than the source's entropy. This necessarily entails some lossiness, or distortion, between the original source data and the best approximation thereto that can be produced on the basis of the encoder's output bits. Rate‐distortion theory was introduced in the seminal works written in 1948 and 1959 by C. E. Shannon, the founder of information theory. We describe Shannon's contribution and then trace its subsequent development worldwide. Heavier than usual emphasis is placed on the concept of “matching” a channel to a source in the rate‐distortion sense, and also on the analogous matching of a source to a channel. Experimental evidence has been mounting in support of the hypothesis that living organisms often simultaneously achieve both of these matchings when processing their sensory inputs, thereby eliminating the need for the complex encoding and decoding operations that are needed in order to produce an information‐theoretically optimum system in the absence of such double matching.