Adapting Segment Anything Model 2 for Few-Shot Semantic Segmentation via Low-Rank Adaptation

Speaker: Bernardo Forni, visiting PhD from Università di Pavia

Abstract: Few-shot semantic segmentation aims to segment unseen classes from only a few annotated samples. Existing approaches learn additional task-specific parameters on top of pretrained models and require extensive training on segmentation datasets. In this talk, we present FS-SAM2: a Few-Shot semantic segmentation method based on the Segment Anything Model 2 (SAM2). SAM2 is a foundational model for promptable image and video segmentation, trained without semantic label supervision. We repurpose SAM2's video memory attention to match features between support and query images. We fine-tune a small subset of parameters using Low-Rank Adaptation (LoRA), specializing the model for semantic correspondence while retaining SAM2's strong segmentation performance. FS-SAM2 achieves remarkable results on standard benchmarks and demonstrates excellent computational efficiency.

Biography: Bernardo Forni is a Ph.D. student in Computational Mathematics at the University of Pavia, in a joint program with Università della Svizzera Italiana (USI).
His research focuses on computer vision and operations research using neural networks, particularly on few-shot model adaptation for semantic segmentation and anomaly detection.

Host: Alessandro Giusti