The International Radio Consultative Committee (CCIR) has developed a testing process that uses human subjects to determine audio quality in digital systems that use lossy compression. Testers play selected recorded material for panels of listeners, who rate the impairments they hear in categories from 5.0 (transparent) to 1.0 (very annoying).
To qualify listeners, testing starts with an "ABX" test procedure. In each trial, the listener is pseudorandomly presented with a selection from known A and B sources and from an unknown X source that can be either A or B. The listener must then identify whether X is the same as A or B. This establishes a baseline for selecting people who can reliably hear differences between sources.
To evaluate the relative audio transparency of some element of the signal chain, a panel of qualified listeners is supplied with test discs that contain a reference track and an arbitrary sequence of test tracks. The panelists are required to compare the other tracks to the reference.
Guidelines in ITU-R Recommendation BS.1116-1 address the selection of audio materials, playback system performance, the listening environment, the assessment of listener expertise, the grading scale, and methods of data analysis. For details, see "Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems," which is downloadable at www.itu.int/rec/R-REC-BS.1116-1-199710-I/en for 22 Swiss francs.
For further details, see "Measurement and Evaluation of Analog-to-Digital Converters Used in the Long-Term Preservation of Audio Recordings," by Ken C. Pohlmann of the University of Miami Frost School of Music, at
www.clir.org/activities/details/AD-Converters-Pohlmann.pdf. Pohlman's paper contains additional information about the statistical evaluation of these subjective measurements, along with more information on how the process is related to analog-to-digital converter (ADC) evaluation. Â