Which objects are the same across these photos?

We want to identify the same physical item when different cameras see it from different angles, amid clutter and similar-looking objects. MessyTable gives us labeled scenes for testing this question.

For this stage, every method receives the same human-annotated object boxes. We compare how well the methods associate those objects across views. All models use frozen pretrained checkpoints.

Full-scene photos + human boxes→Score possible object matches→Associate objects across views→Check labeled identities

The benchmark we kept

100 scenes from 100 different camera setups, 900 photos and 3,600 view pairs. We retain every pair among the nine cameras in each scene. The selection contains 30 easy, 50 medium and 20 hard scenes, chosen by a fixed hash rule.

There are 21,706 annotated object appearances and 78,335 true cross-view object pairs among 2,484,124 possible pairs. A positive match means the same physical item.

This is a 100-scene sample from the official test split. Our older 100-pair experiments have their own data selection and scoring settings.

Preserved scene selection · Official-protocol sample baseline

What we are comparing against

ASNet represents the original appearance-and-surroundings approach. ViT-P3DE Tiny uses a transformer with learned camera and image-position information. The published full-test numbers provide context, and the runs on our fixed sample establish the controls for every variation below.

ModelPopulationAverage precision ↑False-positive rate at 95% recall ↓Fully correct image pairs ↑

AP evaluates the ranking across thresholds. FPR95 measures false matches when recovering 95% of true object pairs. IPAA100 is the fraction of image pairs with every object association and absence decision correct under the authors’ assignment procedure.

Published figures come from the ASNet paper and ViT-P3DE paper. The full-test claims remain unverified by our sampled experiment.

What we changed in matching

DirectionQuestion tested
Object appearance and contextDo tight crops, wider context or multiple crop sizes improve DINOv2?
Foreground and image processingDo SAM masks, GrabCut masks, grayscale or edges improve the object representation?
Orientation and camera geometryDo quarter-turns, shared-view rotation or calibrated geometric constraints improve scores?
Perspective correctionDoes rectifying a validated visible surface improve over the same unwarped surface?
Dense correspondence and fusionCan RoMa pixel correspondence, foreground masks and fixed appearance blends improve the references?

We reuse prior crop scales, rotation angles, geometry rules and fusion weights. Perspective fitting uses foreground keypoints and a held-out correspondence check before rectifying a surface patch. A native-patch control compares the same visible area without the warp.

Change testedControl APModified AP

Top three configurations on this sample

Ranking uses AP first, then lower FPR95, then higher IPAA100. The reference controls are excluded from the three selected variations. These winners were selected after comparing the fixed arms on this sample.

All scored configurations

MethodAP ↑FPR95 ↓IPAA100 ↑Correct linksWrong linksLink recall

AP and FPR95 compare the score rankings across all candidate pairs. The final assignment counts use the authors’ global 95th-percentile distance normalization, original matching solver and fixed 0.70 cutoff. New scorers have fixed distance transforms and received no per-method cutoff fitting. Their IPAA and final assignment counts therefore also reflect score calibration.

What was verified

The sample manifest and human boxes were retained exactly. Every configuration scored the same 2,484,124 candidates and 78,335 positives. The ASNet and ViT reference metrics reproduced the preceding sample runs exactly. The original metric functions and shipped native Linux matching solver ran in a pinned Python 3.7 environment. Crop and dense-feature caches were checked against the original inference paths.

All labels are supplied by the benchmark. Predictions and fixed scoring policies were saved before the sweep’s label-based metric evaluation. The selected winners represent performance on this benchmark, with generalization outside this sample still to be evaluated.

Complete results and per-scene outcomes · Frozen variation plan · Exact score rules and input hashes