9th International Conference on ICT for Sustainable Development

Authors - Thi-Diem Truong, Phuoc-Hai Huynh, Van Hoa Nguyen, Thanh-Nghi Do
Abstract - In disease classification, multi-modal fusion model has emerged as a promising approach to enhance both diagnostic accuracy and efficiency. In this paper, we propose to train the multi-modal model for classifying respiratory illnesses from chest X-ray images and symptom texts. The proposed model uses Support Vector Machine (SVM) on top of Vision Transformer (ViT) and Random Forest (RF) with Term Frequency- Inverse Document Frequency (TF-IDF) for the disease classification. We collect a new real dataset from An Giang province regional general hospital. The experimental results show that using only the ViT model for classifying diseases from chest X-ray images achieves an accuracy of 62.43%. Furthermore, disease classification based on clinical symptom text achieves accuracies of 77.52% and 70.10% respectively when using RF and SVM models with TF-IDF technique. Our multi-modal model achieves an accuracy with 80.10%, surpassing uni-modal models using only images or text.