Machine learning techniques are increasingly being used in medical image analysis. In particular, the CNN (Convolution Neural Network) method, which has shown great successes in object recognitions, has been widely applied to analyze high-levl features from medical images. Unfortunately, owing to the limitation of gathering clinical data, this method has faced obstacles in the clinical invironment: it is difficult to collect sufficient data to guarantee satisfactory classification for various cases of medical images. This problem can be evaded by designing modality specific structured CNN, which takes account of doctor’s decision process, anatomical structure, and the characteristics of the medical image. To get total ent-to-end solutions with practical significance and value, it is necessary to have an in-depth understanding of the underlying physical phenomena with data acquisition systems as well as the details of learning algorithms about how feature representation is created.