Exploring the Limits of Foundation Models in Medical Image Segmentation: A Case Study With SAM and Genetic Algorithms

Abstract

This paper investigates the limits of foundation models in medical image segmentation, mainly focusing on SAM by Meta. While previous research demonstrated SAM’s potential for cost-efficient segmentation, this study explores its performance enhancement through integration with prompt enhancement optimization and genetic algorithms, aiming to minimize user input further. As a proof of concept, we apply this novel approach to lung segmentation tasks using public axial lung CT scans, frontal chest X-ray datasets, and spleen MRIs. Our findings reveal that the genetic algorithm optimization significantly improves SAM’s segmentation accuracy, bringing it closer to the state-of-the-art performance achieved by specifically trained models. In particular, when compared with our previous approach, this technique reaches a 94.85 % Jaccard Index (+3.77 delta) and a 97.17 % Dice Score (+2.50 delta) for lung CT scans, a 93.39 % Jaccard Index (+5.95 delta) and a 96.57 %Dice Score (+3.38 delta) for chest X-rays, and a 91.00 % Jaccard Index (+6.51 delta) and a 95.07 % Dice Score (+4.12 delta) for spleen MRIs. Notably, this improvement is achieved without retraining or modifying SAM’s architecture. However, our analysis also identifies an inherent limitation in this optimization approach, revealing a performance ceiling that cannot be surpassed despite further genetic algorithm iterations. The implications of these findings emphasize the potential of combining foundation models with non-intrusive optimization techniques for cost-effective and accessible medical image segmentation. While dataset-related limitations may affect generalizability, validating the approach across broader clinical scenarios remains essential. Future work should explore applications to additional organs, diverse datasets, and the integration of expert-in-the-loop strategies to enhance clinical utility.