Research project · 2026

SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching

Xu Pan^1,2,†,‡Zhen Pang^1,†Qiyuan Ma¹He Chen¹Wei Ji³Shuhan Shen⁴Xianwei Zheng^1,*

State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University
School of Electrical and Electronic Engineering, Nanyang Technological University
National Key Laboratory of Space Target Awareness, Space Engineering University
Institute of Automation, Chinese Academy of Sciences

^† Equal contribution · ^‡ Work done at LIESMARS · ^* Corresponding author

arXiv preprint, 2026

Abstract

Reliable feature matching becomes brittle when two views share only partial content or depict the same structure at very different scales. SAMatcher takes a region-first approach^[1]: before estimating point correspondences, it predicts consistent co-visible masks and bounding boxes in a shared cross-view representation. Built on SAM-HQ^[2], the framework combines symmetric cross-view interaction with joint mask, box, and mask–box consistency supervision. The resulting semantic and geometric priors suppress unsupported regions and improve classical, sparse learned, and dense matching pipelines under wide baselines.

Motivation for co-visible region modeling — **Fig. 1.** Large scale differences create pixel-level confusion. Explicit co-visible regions retain mutually supported content and constrain matching to geometrically meaningful areas.

Method

SAMatcher framework — **Fig. 2.Overview of SAMatcher.** A shared encoder extracts both views; symmetric fusion exposes cross-view context; prompt-driven mask and box decoders predict complementary co-visible priors that guide correspondence estimation.

Symmetric cross-view fuser — **Fig. 3.Symmetric cross-view feature interaction.** Paired views are treated as interacting token sequences with shared parameters and view offsets, enabling bidirectional alignment without discarding view-specific structure.

Region first

Co-visible masks identify dense mutual support; bounding boxes provide a compact geometric envelope. Both are predicted before matching rather than inferred after correspondences fail.

Symmetric interaction

Bidirectional feature exchange aligns semantics across views while preserving their distinct spatial layouts, allowing co-visibility to emerge as a relation between images.

Joint constraints

Point-sampled mask loss, box regression, and mask–box consistency regularize dense coverage and geometric localization together.

Evaluation

We evaluate on MegaDepth^[3] image pairs selected for large scale differences, reporting pose AUC, accuracy, and mean Average Accuracy (mAA) at multiple thresholds. SAMatcher is attached to nine representative pipelines—from SIFT and nearest-neighbor matching to SuperGlue^[4] and LoFTR^[5]—and compared with both the unmodified pipeline and OETR^[6] region priors.

+12.95 ppLoFTR AUC@20 gain over its base pipeline

+14.70 ppLoFTR mAA@20 gain

78.86 / 82.38full-model mask / box IoU (%)

Pipeline	Prior	AUC@5	AUC@10	AUC@20	Acc@5	Acc@10	Acc@15	Acc@20	mAA@5	mAA@10	mAA@20	P	MS
CON+NN	None	28.07	43.91	58.91	49.80	66.60	74.80	77.80	49.80	58.20	67.25	70.76	9.77
CON+NN	+OETR	30.74	47.70	63.45	52.00	72.40	79.80	84.20	52.00	62.20	72.10	75.66	14.27
CON+NN	+SAMatcher	32.97	50.46	65.79	56.80	73.40	81.20	87.00	56.80	65.10	74.60	79.84	91.31
D2+NN	None	0.85	1.40	2.29	1.27	2.29	3.31	3.82	1.27	1.78	2.67	9.34	2.23
D2+NN	+OETR	7.33	12.04	18.11	13.32	20.34	23.97	27.85	13.32	16.83	21.37	34.18	3.07
D2+NN	+SAMatcher	11.16	19.02	28.37	20.93	31.40	37.91	42.56	20.93	26.16	33.20	49.82	89.99
DISK+NN	None	5.24	7.92	10.36	8.75	11.41	12.47	14.32	8.75	10.08	11.74	18.77	0.82
DISK+NN	+OETR	20.63	30.06	38.10	34.58	43.03	45.52	47.76	34.58	38.81	42.72	56.70	4.88
DISK+NN	+SAMatcher	27.78	40.71	53.36	45.52	59.34	66.75	69.57	45.52	52.43	60.29	75.09	89.65
DISK+SG	None	25.29	37.49	50.06	43.37	55.82	62.45	68.27	43.37	49.60	57.48	39.50	9.62
DISK+SG	+OETR	29.66	44.45	58.74	50.80	66.40	73.00	78.20	50.80	58.60	67.10	69.48	15.81
DISK+SG	+SAMatcher	32.27	48.14	62.79	54.60	70.80	77.20	82.80	54.60	62.70	71.35	78.39	62.82
SIFT+NN	None	5.85	11.13	19.79	11.20	20.60	28.80	35.00	11.20	15.90	23.90	17.06	85.93
SIFT+NN	+OETR	10.05	16.91	26.47	18.20	28.80	36.60	41.40	18.20	23.50	31.25	23.32	85.40
SIFT+NN	+SAMatcher	10.64	20.64	34.02	22.60	37.80	47.20	55.20	22.60	30.20	40.70	27.62	93.61
R2D2+NN	None	20.47	33.45	47.27	37.60	52.60	61.60	68.00	37.60	45.10	54.95	52.89	87.81
R2D2+NN	+OETR	30.34	46.03	62.22	51.40	70.00	79.20	83.60	51.40	60.70	71.05	73.59	91.95
R2D2+NN	+SAMatcher	34.12	52.54	69.07	59.80	78.60	86.00	90.00	59.80	69.20	78.60	82.59	97.23
SP+NN	None	3.74	5.81	9.64	5.60	10.40	13.40	17.00	5.60	8.00	11.60	17.31	3.94
SP+NN	+OETR	13.43	21.74	31.63	24.60	35.60	41.60	45.80	24.60	30.10	36.90	40.11	9.54
SP+NN	+SAMatcher	22.53	36.18	50.42	40.80	56.60	65.60	70.20	40.80	48.70	58.30	60.62	90.61
LoFTR	None	34.36	48.33	60.79	53.80	68.20	73.20	77.20	53.80	61.00	68.10	55.61	43.91
LoFTR	+OETR	36.64	53.04	67.30	61.00	75.40	81.60	85.80	61.00	68.20	75.95	74.12	50.06
LoFTR	+SAMatcher	42.34	58.93	73.74	66.20	83.40	89.00	92.60	66.20	74.80	82.80	93.34	55.11
SP+SG	None	33.46	49.64	64.75	55.20	71.60	80.80	85.20	55.20	63.40	73.20	85.19	13.54
SP+SG	+OETR	35.69	51.89	66.36	58.00	74.80	80.60	86.20	58.00	66.40	74.90	90.41	22.66
SP+SG	+SAMatcher	36.41	54.43	69.97	62.60	79.40	85.20	90.40	62.60	71.00	79.40	94.96	79.49

Table 1. Complete MegaDepth evaluation under large scale differences (%). All nine feature–matcher pipelines are reported without a prior, with OETR, and with SAMatcher.

Variant	Mask IoU (%)	Box IoU (%)
w/o symmetric interaction	75.06	71.90
w/o joint constraint	76.67	80.68
w/o view offset	77.15	80.74
Full SAMatcher	78.86	82.38

Table 2. Ablation on MegaDepth. Symmetric interaction contributes most strongly to box localization, while joint constraints and view offsets provide complementary gains.

Results

The quantitative gains are consistent across descriptor families, but the visual evidence explains why: SAMatcher learns mutual visibility rather than image saliency, then uses this support to remove structurally invalid correspondences.

Relative performance gains — **Fig. 4.** Ridge-style relative gains on MegaDepth. SAMatcher improves every evaluated matching configuration more consistently than the box-only OETR prior.

Co-visible masks — **Fig. 5.** OETR box predictions versus SAMatcher masks. The learned masks follow mutually visible content under large viewpoint changes and partial overlap.

Region-guided matching — **Fig. 6.** Region-guided correspondence comparison. Green and red lines denote correct and incorrect matches; SAMatcher retains valid support where coarse boxes are missing or inaccurate.

Mask and box co-refinement — **Fig. 7.** Masks favor recall and boxes provide compact localization. Their intersection suppresses over-extended predictions and yields a more reliable co-visible region.

**Fig. 8.** Zero-shot generalization on unseen GL3D (top) and ScanNet (bottom). The model transfers co-visibility reasoning across outdoor aerial and indoor domains without fine-tuning.

Citation

@article{pan2026samatcher,
  title={SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching},
  author={Pan, Xu and Pang, Zhen and Ma, Qiyuan and Chen, He and Ji, Wei and Shen, Shuhan and Zheng, Xianwei},
  journal={arXiv preprint arXiv:2606.03406},
  year={2026}
}

References

X. Pan, Z. Pang, Q. Ma, et al. “SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching.” arXiv:2606.03406, 2026.arXiv
L. Ke, M. Ye, M. Danelljan, et al. “Segment Anything in High Quality.” NeurIPS, 2023.arXiv
Z. Li and N. Snavely. “MegaDepth: Learning Single-View Depth Prediction from Internet Photos.” CVPR, 2018.arXiv
P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. “SuperGlue.” CVPR, 2020.arXiv
J. Sun, Z. Shen, Y. Wang, et al. “LoFTR: Detector-Free Local Feature Matching with Transformers.” CVPR, 2021.arXiv
H. Chen, Z. Luo, L. Zhou, et al. “A Span-Based Object-Level Region Proposal Network for Scene Matching.” CVPR, 2022.arXiv