MIND's real exploration: an order that is not one¶
The order recorded in MIND is indistinguishable from a shuffle
The within-feed exchangeability test gives \(z = +0.12\) (\(p = 0.91\)) across the 156,965 feeds of the training split, and \(z = +0.28\) (\(p = 0.78\)) across the 73,152 of the second. Clicks fall exactly where chance would put them — in a dataset where the same test would detect \(\eta = 0.02\) at twelve standard deviations.
And the curve one naturally plots sees a textbook position bias in it
Click-through rate by position falls from 0.108 to 0.038 over the first twenty ranks, and the log-log fit gives \(\hat\eta = 0.39\). It is a composition artefact: position 20 only exists in long feeds, where the click rate per item is mechanically lower. At fixed feed length, the curve is flat.
Correcting the previous stage: the identifiability check is not enough
estimate_position_bias refuses to estimate \(\eta\) when no item has changed rank. In MIND,
thousands of items changed rank — artificially. The estimator therefore answers, and its
answer is an increasing function of a mere nuisance parameter: from \(-0.13\) to \(+0.25\)
depending on the impression threshold, never once ceasing to be "significant".
What this measurement had to settle¶
Adversarial rank established two things: the severity \(\eta\) of position bias can be estimated rather than posited, and positing it wrongly costs up to 179 % on the published figure. It also established the condition under which that estimate exists — that the platform did not always rank the same items in the same places.
That condition is a property of the dataset, not of the method. Hence the prerequisite written into roadmap §3.1: measure the real exploration before evaluating anything on it. MIND (Microsoft News Dataset, Wu et al., ACL 2020) is the reference dataset for news recommendation, and the one on which the algorithm was to be evaluated.
The aggregate curve and its refutation at fixed length; what the test would have detected; the five severities drawn from the same dataset; and what the destroyed variable costs. Figure regenerated by notebook 16.
1. The identifiability condition is met — abundantly¶
| Quantity | Value |
|---|---|
| feeds | 156,965 |
| items served | 5,843,444 |
| distinct items | 20,288 |
| items seen at several ranks (threshold 5) | 2,655 |
| distinct ranks per item (median) | 16 |
| highest rank observed | 299 |
A median retained item was served at sixteen different positions. No real dataset could offer better: on this count, exploration looks ideal.
2. The curve one naturally plots¶
| Rank | 1 | 2 | 3 | 5 | 10 | 15 | 20 |
|---|---|---|---|---|---|---|---|
| click rate | 0.108 | 0.108 | 0.081 | 0.066 | 0.050 | 0.043 | 0.038 |
A regular decay over 5.8 million items served, fitted at \(\hat\eta = 0.39\). It is the first plot anyone looking for position bias produces, and it has exactly the expected shape.
3. The confounder¶
MIND's feeds are not all the same length — median 24, mean 37, maximum 299. But position 20 only exists in feeds of at least 20 items, and the click rate per item served is mechanically lower there: a reader who clicks once in a feed of 100 items produces a rate of 0.01; the same click in a feed of 4 items produces 0.25.
Holding length fixed separates the two effects.
| Feed length | 6 | 10 | 15 | 20 | 30 | 50 |
|---|---|---|---|---|---|---|
| apparent \(\eta\) | −0.022 | +0.025 | −0.025 | −0.033 | −0.002 | +0.037 |
The slope is zero, and its sign changes from one length to the next. The 0.39 of the aggregate curve measured nothing but the mixture of lengths.
4. The exact test, and what it would have detected¶
The fixed-length control is telling but partial: it discards most feeds and still lets the mean quality of a feed's items, or its reader's appetite for clicking, play a role. The exact test conditions on the feed.
For a feed of length \(L\) carrying \(k\) clicks, sum the normalised ranks \(u_R = (R - 1/2)/L\) of the clicked items. Under the hypothesis that clicks are indifferent to position, those \(k\) positions are a draw without replacement from the feed's \(L\) positions, whose expectation and variance are known exactly:
Position bias concentrates clicks at the top: it makes the standardised deviation negative.
| Split | Feeds | Observed sum | Expected | \(z\) | \(p\) |
|---|---|---|---|---|---|
| train | 156,965 | 118,187.9 | 118,172.0 | +0.116 | 0.908 |
| dev | 73,152 | 55,717.7 | 55,691.5 | +0.278 | 0.781 |
A test that rejects nothing says nothing until one knows what it would have rejected. The calibration runs on simulated logs with the same feed structure as MIND, under a position bias of known severity.
| simulated \(\eta\) | 0.00 | 0.02 | 0.05 | 0.10 | 0.25 | 0.50 | 1.00 |
|---|---|---|---|---|---|---|---|
| \(z\) | +0.9 | −9.0 | −27.0 | −49.2 | −104.4 | −161.2 | −199.3 |
The minimum detectable severity is \(\eta \approx 0.004\). MIND's recorded order therefore carries no placement information beyond that threshold.
The dataset's documentation did say so, in one line: "the orders of news in a impressions have been shuffled". A line of documentation, however, says neither what remains of it nor what the measurement returns when one ignores it.
5. Five estimates from the same dataset, four too many¶
| Method | \(\hat\eta\) | Standard error |
|---|---|---|
| naive aggregate fit | +0.388 | — |
| fixed effects, threshold 5 | −0.131 | 0.002 |
| fixed effects, threshold 10 | −0.047 | 0.002 |
| fixed effects, threshold 20 | +0.053 | 0.003 |
| fixed effects, threshold 50 | +0.194 | 0.004 |
| fixed effects, threshold 100 | +0.255 | 0.007 |
| exchangeability test | 0.000 | ±0.004 |
A negative severity would mean that lower positions receive more clicks. Every one of these estimates carries a standard error below 0.007 — hence all "significant", and all mutually incompatible.
This is the demonstration that the identifiability check of adversarial rank is necessary and not sufficient: it counts rank variation without saying where it comes from, and artificial variation satisfies it better than any real exploration.
6. Shuffling does not debias the clicks¶
A widespread reading holds that shuffling the recorded order protects against position bias. It conflates two things.
MIND's clicks were produced by readers looking at an ordered feed — Microsoft News's real order, the one the dataset did not keep. They therefore carry position bias in full. What the shuffle removed is the rank, the only regressor that would have allowed one to account for it.
On a simulated log whose truth is known (\(\eta = 1.00\)):
| Log | \(\hat\eta\) |
|---|---|
| order kept | 1.003 ± 0.008 |
| order shuffled within each feed | −0.003 ± 0.006 |
The clicks are the same in both rows. Only the variable that allowed them to be corrected has gone.
What that loss costs the evaluation¶
| Estimated cost of a diversity filter | Value |
|---|---|
| true cost | 6.61 % |
| with \(\eta\) read off the kept order | 6.68 % |
| with \(\eta\) read off the shuffled order | 0.00 % |
The zero is not a numerical coincidence. Under \(\eta = 0\), every position is deemed equally seen: two policies differing only in order then receive the same estimated value, whatever they do. The evaluation is not slightly wrong — in this setting it is vacuous by construction, and its verdict is that re-ranking costs nothing.
7. What this measurement settles¶
MIND cannot calibrate \(\eta\). This is no defect of the method nor a shortage of data — 5.8 million items served — but the absence of the one variable that would identify severity. No refinement of the estimator will change it.
For this repository. Evaluating the algorithm on MIND remains possible for whatever does not depend on exposure: feed composition, diversity served, cost in declared relevance. It is impossible for whatever does — the counterfactual estimation of engagement cost, which was the point of the exercise. Either a dataset that records display rank is needed, or an imported \(\eta\) must be assumed, whose price adversarial rank has quantified.
For anyone evaluating a re-ranking on public data. The check to run is not "do I have enough rank variation?" but "does that variation come from the platform or from the anonymisation?". The two look identical from the estimator's point of view, and only the second produces confident and false figures.
For those who publish logs — and for the regulator who demands them. Shuffling the display order does not make a dataset unbiased: it makes it uncorrectable. Publishing the served rank, or failing that the exposure propensity, costs one column and decides what remains measurable. → memorandum
Data provenance¶
MIND-small's official link (mind201910small.blob.core.windows.net) now answers 409 Public
access is not permitted on this storage account: the dataset is no longer downloadable from its
source. The logs used here come from a mirror, and a mirror is to be verified: feed counts
matching the statistics published with MIND-small (156,965 and 73,152), SHA-256 digests recorded
in ide.mind.
The raw dataset is not versioned — Microsoft Research licence, 135 MB. The repository carries a
1.5 MB digest, data/mind_digest.npz, retaining the feeds' order structure and the
sufficiently observed (item, rank) cells. It reproduces every figure on this page identically,
which a test verifies, and is rebuilt by scripts/fetch_mind.py then
scripts/build_mind_digest.py.
Reservations¶
The exchangeability test bears on the recorded rank. It establishes that this rank does not predict clicks; it says nothing about the order actually displayed, which is not in the dataset and cannot be recovered from it. That is precisely the limit of the result: Microsoft News's position bias remains unknown, and unknowable from MIND.
The power calibration further assumes the model \(e(R) = R^{-\eta}\). A cascade examination model, or one depending on the item, would produce a different form of rank dependence — which the test would detect just as well, since it only tests independence, but which the minimum detectable severity would no longer summarise.
Open leads¶
- ~~Find a public dataset that records the served rank.~~ → done: Baidu-ULTR records it and the test rejects there at \(z = -206\); the Open Bandit Dataset additionally publishes true propensities. Neither carries a usable viewpoint label, so the transposition is not free.
- Evaluate on MIND whatever does not depend on exposure, and say so: composition, diversity served, trade-off frontier in declared relevance.
- Make the exchangeability test an acceptance check for logs supplied under Article 40 of the DSA: a log whose order is indistinguishable from a shuffle is not auditable counterfactually, and it is better to know that before the audit than after.
- Compare against tuned baselines — MMR, random re-ranking, popularity — which remains the oldest debt of the evaluation programme.
Implementation: ide.mind, scripts/fetch_mind.py, scripts/build_mind_digest.py ·
Notebook: 16 — MIND's real exploration ·
adversarial rank · rank and counterfactual ·
roadmap