Comparative analysis of data collection methods for individualized modeling of radiologists’ visual similarity judgments

Abstract

Rationale and Objectives: We conducted an observer study to investigate how the data collection method affects the efficacy of modeling individual radiologists’ judgments regarding the perceptual similarity of breast masses on mammograms. Materials and Methods: Institutional review board approval was obtained prior to the study. Six observers of variable experience levels in breast imaging were recruited to assess the perceptual similarity of mammographic masses. The observers’ subjective judgments were collected using: (i) a rating method, (ii) a preference method, and (iii) a hybrid method combining rating and ranking. Personalized user models were developed with the collected data to predict observers’ opinions. The relative efficacy of each data collection method was assessed based on the classification accuracy of the resulting user models. Results: The hybrid data collection method produced significantly more accurate individualized user models of perceptual opinions with comparable and sometimes better time efficiency than the other two data collection methods. The user models derived from hybrid data were clearly superior even when developed with a dramatically smaller number of training cases. Conclusions: A hybrid method combining rating and ranking is an intuitive and efficient way for collecting subjective similarity judgments to model human perceptual opinions with a higher accuracy than other more commonly used data collection methods.