Updated with notes from Dominik’s talk on 10/14

website demo is really convincing https://dominik-schnaus.github.io/unpaired-rosetta/

align text embeddings from Qwen3-8B and image embeddings from DINOv2 without any labeled image-caption pairs!

Math / Figure 2

This is the main objective they’re trying to optimize:

We want two things:

$W$, a $d_X \times d_Y$ mapping from the text embedding space to the vision embedding space
$T$, where $T_{ij}$ is $1$ if text data point $i$ maps to image data point $j$ and $0$ otherwise.

Right: initialization of candidates for $W$ and $M$:

We first want a reasonable initialization for $W$
1. (line 6) We do k-means clustering on both text space and image space separately.
2. (line 7) Then, we pair clusters in text and vision space to each other by checking through all permutations $P$ (in practice, they do this using some quadratic programming) of the clusters on one side and finding the permutation with highest CKA similarity between the clusters (CKA similarity depends on the ordering of the clusters), which ensures that the similarity between cluster $\ell$ and cluster $r$ in text-space should be close to the similarity between cluster $\ell$ and cluster $r$ in corresponding clusters in image space.
3. (line 8) Using this pairing $P$, we construct the first candidate for $W$, which they call $M$.
  1. if you just do one clustering it’s possible that the clusters don’t match up exactly (one cluster might not have an equivalent in the other space), hence why they needed to average over many clusterings.
  2. the batching is just for computational efficiency of the matching, can only match b to b points at a time
Freezing this $W$, they construct the first candidate for $T$. By the above, $XMY^T$ is the similarity scores $s_{ij}$ between all points $i$ and $j$. Hungarian matching then solves for the optimal matching $T$ trying to minimize the sum of the similarity scores $s_{ij}$ between matched points.

Left: Wasserstein Procrustes to iteratively improve $W$ and $M$

Line 5/9: If you freeze $T$, you can solve for the closed form $W$ which optimizes the equation (1) objective using some SVD

Line 8: now freeze $W$ and then solve for the best $T$, the same as we did in Right panel step
Repeatedly refine $W$ and $T$. For the refinement part, you don’t even need to take an EWMA of the W, it stabilizes on its own as you keep taking batches and rewriting the entire $W$ using just the current batch.

Experiments

Method details

used last hidden state for all embeddings (didn’t ablate this), tried 3 different token poolings (sophie’s, mean pooled over tokens, qwen3-emb EOS token)

k=30 chosen just because that’s what their NP opt solver can do, would want more clusters if possible

They use FOS-CTTM as their main metric: if you take a text embedding $x$ and and map it using $W$ to an image embedding $y$, and you have the ground truth pairings between e.g. images $x$ and their captions $y$, then out of all the images, what fraction of them are closer to $Wx$ than $y$ is? Ideally, this number is $0$

(a) in the first column, they train on unpaired captions and images from different datasets, and then just validate the learned $W$ using matched image-caption pairs on MS COCO.
(c) if the actual image-caption pairs are more CKA aligned, the method works better
dinov2 much better than dinov3 — maybe the embedding extraction should be different
qwen3 generative embeddings worse with long captions — embedding model already has enough to work with, don’t need to generate

works on bio model spaces to bio model spaces