Abstract:The incidence of breast cancer continues to rise worldwide. Due to the heterogeneity of breast cancer, there is an overlap between benign and malignant tumor images that only using a type of images cannot obtain satisfactory classification results. This paper proposes a two-stage breast cancer diagnosis system based on ultrasound and mammogram images. In the first phase, an abstaining classification method is used to classify breast ultrasound (BUS) images, in which some BUS tumors are classified with high confidence, and the uncertain tumors are not classified. These unclassified tumors are then classified using mammogram images in the second stage. Supplemented by mammogram information, the system can diagnose breast cancer by utilizing multimodal image information to screen for unrecognizable ultrasound images. The ultrasound and mammogram images used in this study were provided by the Cancer Hospital of Harbin Medical University and the First Affiliated Hospital of Harbin Medical University. The abstaining method and the two-stage diagnosis system were validated in experiments. Compared with diagnostic systems using only BUS features, the proposed two-stage diagnostic system provided better performance with the accuracy of 92.59%, AUC of 0.933 3, G-mean of 93.09%, sensitivity of 86.67%, specificity of 100%, positive predictive value of 100%, negative predictive value of 85.71%, and Matthew’s correlation coefficient of 0.861 9. Experimental results demonstrate that adding mammogram information can increase the performance of the diagnosis system that uses BUS images only.