Two logs that record the served rank¶
The positive control passes, and on the right side
On Baidu-ULTR, the exchangeability test that detected nothing in MIND rejects at \(z = -206\) (\(p < 10^{-12}\)). The sign matters as much as the magnitude: clicks concentrate at the top, which position bias requires and a shuffle does not produce.
Severity no longer has to be posited
\(\hat\eta = 1.10 \pm 0.09\) by document fixed effects on a search results page. The aggregate fit gives 1.49: the gap is the quality confounder, the platform placing the best documents on top.
But \(\eta\) is a property of the surface, not a constant
On the Open Bandit Dataset, where item-to-position allocation is random — so the effect is causal, with no model — a three-thumbnail horizontal banner decays as \(R^{-0.04}\) to \(R^{-0.11}\). An order of magnitude separates the two surfaces, which is exactly the size of error a transported \(\eta\) costs.
And none of these datasets permits the announced evaluation
That would need the served rank and an interpretable viewpoint label. MIND has the categories without the rank; Baidu-ULTR has the rank without labels; the Open Bandit Dataset has rank, propensity and three categorical attributes — anonymised. The dataset that would allow the algorithm to be evaluated end to end does not exist publicly.
What this search had to find¶
MIND's real exploration ended on a requirement rather than a result: a dataset is needed that records display rank, failing which exposure is not identifiable and the counterfactual evaluation of a re-ranking is vacuous by construction.
Two public logs answer, and not to the same requirement.
| MIND | Baidu-ULTR | Open Bandit Dataset | |
|---|---|---|---|
| served rank recorded | no — shuffled | yes, 1 to 20 | yes, 3 positions |
| feeds grouped | yes | yes, by session | no, one row per impression |
| true propensity published | no | no | yes |
| uniformly random bucket | no | no | yes |
| viewpoint label | yes, editorial | no | anonymised attributes |
| size used here | 5.8 M rows | 0.5 M rows | 5.9 M rows |
| licence | Microsoft Research | CC BY-NC 4.0 | CC BY 4.0 |
What a genuinely served rank does to the curve; the check that tells an order from a shuffle; the position effect measured without a model on a three-thumbnail banner; and the counterfactual estimator judged against the value it estimates. Figure regenerated by notebook 17.
1. Baidu-ULTR: the positive control¶
A slice of 524,164 documents served, 64,200 search sessions, ranks 1 to 20.
| Log | Usable feeds | \(z\) | \(p\) | Reading |
|---|---|---|---|---|
| MIND | 156,965 | +0.12 | 0.91 | order indistinguishable from a shuffle |
| Baidu-ULTR | 29,916 | −205.72 | \(< 10^{-12}\) | the order carries the placement |
This control mattered as much as the verdict on MIND: a test that never rejects anything would be indistinguishable from a blind one. This one rejects, massively, and on the side theory prescribes.
Click rate follows the expected decay: 0.340 at rank one, 0.088 at rank three, 0.010 at rank ten.
2. Severity, measured three ways¶
| Method | \(\hat\eta\) |
|---|---|
| aggregate fit (ranks 1-10) | 1.494 |
| at fixed session length (10 documents) | 1.144 |
| document fixed effects | 1.099 ± 0.089 |
This figure assumes a shape, and the shape is not tested
\(\hat\eta\) means something only if examination follows a power law. Under a cascade model — where the reader stops on finding what they wanted, the home ground of a results page like this one — decay is geometric, and the fitted law overstates real exposure by a factor of 40 to 4,110 at rank twelve. The exchangeability test says the order carries information; it does not say in what form. → The two blind spots
Figure revised since. The same quantity, measured on display rather than estimated through clicks, is \(0.882 \pm 0.046\) over 143 documents. The click-based estimate overstates decay by 23 %, because a click conflates examination with attractiveness. → Measured exposure
On MIND, the gap between the aggregate fit and fixed effects came from the mixture of lengths. Here it comes from something else, and more expected: the platform places the best documents on top, so part of the decay is quality, not exposure. Fixed effects remove it by comparing a document only with itself.
Coverage is thin, and the row count does not say so
Of 444,709 distinct documents, 335 reach the impression threshold and 55 appear at more than one rank. On a results page, the same URL rarely reappears. The estimate is identified — the standard error says so — but it rests on fifty-five documents, not on half a million rows.
3. The Open Bandit Dataset: position measured without a model¶
The random bucket allocates items to positions by chance: the click-rate difference between positions is a causal effect of position, with no examination model to posit. It is the cleanest possible measurement of \(\eta\) — on an entirely different surface, three horizontal thumbnails from left to right.
| Campaign | left | centre | right | \(\hat\eta\) |
|---|---|---|---|---|
| all (1.37 M impressions) | 0.00354 | 0.00347 | 0.00340 | 0.037 |
| men (0.45 M impressions) | 0.00538 | 0.00524 | 0.00475 | 0.105 |
Standard errors are 0.00009 (all) and 0.00019 (men): the effect goes the right way and stays, thumbnail to thumbnail, on the order of two standard errors. It is measurable as a trend, not conclusive position by position — and ten times weaker than on a results page.
Adversarial rank quantified what a wrongly posited \(\eta\) costs: up to +179 %. Transporting the value of a results page onto a banner — or the reverse — is an error of exactly that size.
4. The confrontation: an estimator judged against the truth¶
The Open Bandit Dataset contains two buckets served in parallel by two different policies. So one can do what no other dataset in this repository allows: estimate the value of the uniform policy from the data of a different policy alone, then compare it with its value measured where it actually served.
Ground truth: 0.005124 ± 0.000106, measured over 452,949 impressions of the random bucket. Logging bucket: 4,077,727 impressions served by Bernoulli Thompson Sampling.
| Estimator | Value | Error |
|---|---|---|
| naive (observed click rate) | 0.006743 | +31.6 % |
| IPS | 0.005253 | +2.5 % |
| SNIPS | 0.004768 | −7.0 % |
| IPS clipped at 10 | 0.004473 | −12.7 % |
| IPS clipped at 100 | 0.005238 | +2.2 % |
The correction works. This is the first time in this repository that a counterfactual estimator is judged against the quantity it claims to estimate, rather than against a simulation.
And the diagnostic that forbids declaring victory
The effective sample size is 1,513 for 4,077,727 impressions, i.e. 0.04 %. The unbiased estimate rests on the equivalent of fifteen hundred observations: at a click rate of 0.005, that leaves a margin on the order of half the figure. That the error lands at 2.5 % therefore owes as much to luck as to method.
This is exactly the diagnostic ide.offpolicy requires to be published alongside the
figure, and here is the real case that shows why: without it, one would think one held a
measurement over four million rows.
Clipping is not free either. Capping weights at 10 brings the estimate to −12.7 %: the bias traded for variance is of the same order as the error one set out to correct. The cap must be published with the figure.
5. What this settles, and what it does not¶
Settled — the criterion for choosing a dataset. Neither size nor the presence of editorial labels: the recording of the served rank, which can be verified before any measurement.
Settled — severity can be measured. 1.10 ± 0.09 on a results page, a tenth of that on a three-thumbnail banner. What remains forbidden is transporting one to the other.
Settled — counterfactual estimators hold up against a ground truth, with the quantified caveat that their effective precision is far below what the row count suggests.
Not settled — evaluating the algorithm. That needs the served rank and an interpretable viewpoint label. None of the three datasets measured carries both. A diversity computed over anonymised categories says nothing: the index's reference catalogue is a political declaration, and a declaration over hashes is not one.
This is the third negative result in the series, and it deserves stating as plainly as the other two: the dataset that would permit the announced evaluation does not exist publicly. What remains possible fits in three lines, and the roadmap inherits them:
- measure on MIND whatever does not depend on exposure, and say so;
- measure on Baidu-ULTR and the Open Bandit Dataset whatever does not depend on viewpoints;
- for the rest, ask for the data — which, under Article 40 of the DSA, is a provided-for procedure and not a wish. → the request, drafted · memorandum
Data provenance¶
Baidu-ULTR: slice part-0_split-0 (0.9 GB) redistributed by the University of Amsterdam for
the reproducibility study of Hager et al. (SIGIR 2024), under CC BY-NC 4.0. Four columns are read
of the file's twenty-nine — session, rank, click, document identity; the embeddings, which make
up all its weight, are not.
Open Bandit Dataset: three buckets (2.2 GB) under CC BY 4.0 — the "all" campaign under the random policy, and the "men" pair random / Bernoulli TS, without which the confrontation would have nothing to compare against.
No raw file is versioned. The repository carries a 0.7 MB digest,
data/exposure_digest.npz, reproducing every figure on this page identically — a test verifies
it — and rebuilt by scripts/fetch_exposure.py then scripts/build_exposure_digest.py. Size and
SHA-256 digest of each source are recorded in ide.exposure.
A defect in the digest, found by Baidu-ULTR
MIND's digest summarised a feed's structure by its length, assuming a feed of length \(L\) occupies ranks 1 to \(L\). True of MIND and false of a results page, which skips ranks. The digest now verifies this instead of assuming it, and keeps served ranks one by one when the assumption fails. The defect was painless — it returned figures of the right order, \(z = -200\) instead of \(-206\) — which is how this kind of error survives.
Reservations¶
A slice of Baidu-ULTR is not Baidu-ULTR: 0.5 million rows out of 1.2 billion sessions. Nothing guarantees that severity there is that of the whole dataset, nor of another engine.
The Open Bandit Dataset measures a fashion recommendation on three thumbnails. Transporting its figures to a news feed is exactly the operation this page warns against.
Finally, the confrontation in section 4 concerns the value of a uniform policy, the only one this dataset contains a bucket for. Nothing guarantees IPS behaves as well for a target policy further from the logging one — on the contrary, effective size would be smaller still.
Implementation: ide.exposure, ide.logs, scripts/fetch_exposure.py,
scripts/build_exposure_digest.py · Notebook:
17 — Two logs that record the served rank ·
MIND's real exploration · adversarial rank and severity ·
roadmap