Dual-encoder vision–language models (VLMs) expose a similarity interface that enables zero-shot retrieval but fails compositional constraints: queries like “umbrella and no person” retrieve images containing both, even when concept detection is reliable. We trace this to an interface-level Bag-of-Concepts effect, where similarity scores approximate mean pooling of concept evidence regardless of operators; although operator-dependent signals exist in text embeddings, they are too weak or misaligned to affect rankings. Fine-tuning does not reliably resolve this failure because the dominant bottleneck is how similarity aggregates evidence rather than what encoders represent. We propose factored inference, which separates evidence extraction from constraint execution, and introduce LCSE (Logic-Constrained Score Editing), a training-free method that executes constraints externally using concept scores from frozen encoders. We also introduce FACTOR-Bench, where LCSE achieves 85.5% accuracy versus 73.2% for the best fine-tuned baseline, 90.7% when applied to SigLIP 2, and improves NegBench COCO MCQ accuracy from 27.2% to 65.2% while preserving retrieval performance.
@inproceedings{alshehri2026similarity,title={Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models},author={Alshehri, Sultan and Yang, Zhantao and Zhang, Han and Savvides, Marios},booktitle={International Conference on Machine Learning (ICML)},year={2026},}
NeurIPS 2026 Med‑Reasoner Workshop
Rejected When Asked, Followed When Assumed: False Presuppositions in Medical Vision-Language Models
Sultan Alshehri, Almaan Khan, Sarah Alharbi, Marios Savvides
Paper forthcoming
2020
IEEE Access 2020
Voxel Weight Matrix-Based Feature Extraction for Biomedical Applications
Fahad Albalawi, Sultan Alshehri, Abderrazak Chahid, Taous-Meriem Laleg-Kirati
@article{albalawi2020voxel,title={Voxel Weight Matrix-Based Feature Extraction for Biomedical Applications},author={Albalawi, Fahad and Alshehri, Sultan and Chahid, Abderrazak and Laleg-Kirati, Taous-Meriem},journal={IEEE Access},volume={8},pages={121451--121459},year={2020},doi={10.1109/ACCESS.2020.3006521},}