The counter-expertise's two blind spots¶
The exchangeability test holds under cascade
Tested against a click model where examination depends on what precedes it, the test rejects between \(z = -208\) and \(z = -241\) depending on the continuation probability — more strongly than on Baidu-ULTR (\(-206\)), even though cascade reduces the number of usable feeds. Its negative verdict on MIND therefore keeps its meaning.
But the power law \(R^{-\eta}\) does not hold
Under cascade, attention decays geometrically, not polynomially. At rank twelve, the fitted law overstates real exposure by a factor of 40 to 4,110 — and the estimator returns \(\hat\eta \approx 0.5\) with a standard error of \(0.04\): a perfectly confident figure describing no real exposure.
And under estimated relevance, the tuned methods' advantage fades
The ranking of re-rankers holds at every noise level, but the gap to random selection falls from 16.4 points at zero noise to 5.8 points at \(\sigma = 0.4\), and inverts at \(\sigma = 0.6\). Random is the only method that does not use relevance: its shortfall is invariant.
Why this page¶
The counter-expertise ended by stating what it had not done. These are the repository's two deepest assumptions — one about the shape of attention, the other about what a platform knows of its reader — and neither had been tested.
The exchangeability test under cascade; real exposure against the fitted law; the two shapes of click-rate decay; and the collapse of the tuned methods' advantage when relevance is estimated. Figure regenerated by notebook 21.
1. The test survives, the power law does not¶
The repository uses a position model throughout: examination probability is \(R^{-\eta}\) and depends on rank alone. The cascade model posits something else entirely: the reader moves down the feed, stops on finding what they wanted, and otherwise continues with probability \(\gamma\). Attention decay is no longer a postulated law but a consequence of the traversal.
| Log | \(z\) | Verdict |
|---|---|---|
| MIND (shuffled order) | +0.12 | does not reject |
| Baidu-ULTR (real data) | −205.7 | rejects |
| cascade, \(\gamma = 1.00\) | −240.8 | rejects |
| cascade, \(\gamma = 0.85\) | −226.9 | rejects |
| cascade, \(\gamma = 0.60\) | −207.9 | rejects |
The test assumes nothing, which is why it holds: it tests only independence between position and click. Had it detected only one particular form of rank dependence, its silence on MIND would have distinguished nothing — a shuffled log from a log whose dependence takes another form.
The estimator, by contrast, assumes a great deal. Simulation gives here the ground truth real data never provides: the positions actually examined.
| Rank | Real examination (\(\gamma = 0.85\)) | Fitted \(R^{-\hat\eta}\) | Error |
|---|---|---|---|
| 1 | 1.000 | 1.000 | 0 % |
| 2 | 0.573 | 0.710 | +24 % |
| 4 | 0.190 | 0.504 | +165 % |
| 8 | 0.020 | 0.358 | +1,715 % |
| 12 | 0.0023 | 0.293 | +12,781 % |
A power law cannot approximate an exponential over twelve ranks: it fits at the top and diverges at the tail. And the fit looks excellent — \(\hat\eta = 2.19\) over the first ten ranks, standard error \(0.04\) by fixed effects — on a shape that is not the right one.
What this takes away from publishing \(\hat\eta\)
The repository publishes \(\hat\eta = 1.10 \pm 0.09\) on Baidu-ULTR. That figure means something only if examination there follows a power law. Nothing guarantees it: Baidu-ULTR is a search results page, the cascade model's home ground, and Hager et al. (2024) report precisely that a misspecified position model would explain their negative result.
Severity must therefore be published with its assumed shape, and that shape remains to be tested. The exchangeability test says whether the order carries information; it does not say in what form.
2. What estimated relevance does to the comparisons¶
The baselines compare five re-rankers against the exact frontier, and assume relevance known throughout. No platform is in that position.
The protocol is changed in one respect only: methods rank on perceived relevance and are judged on the true one.
| Noise \(\sigma\) | Rank correlation | Filter | MMR | Round robin | Boltzmann | Random |
|---|---|---|---|---|---|---|
| 0.00 | 1.000 | 0.0 % | 0.0 % | 3.2 % | 2.2 % | 16.4 % |
| 0.10 | 0.899 | 1.3 % | 2.2 % | 4.7 % | 3.8 % | 16.4 % |
| 0.20 | 0.751 | 5.0 % | 6.0 % | 7.4 % | 7.7 % | 16.4 % |
| 0.40 | 0.500 | 10.6 % | 10.5 % | 13.6 % | 12.9 % | 16.4 % |
| 0.60 | 0.359 | 15.4 % | 15.2 % | 16.6 % | 16.9 % | 16.4 % |
The ranking of methods does not change — filter and MMR ahead, random behind. The baselines' conclusion is not invalidated.
But the advantage collapses. The reason is mechanical: random selection is the only re-ranker that does not use relevance, so its shortfall is invariant to noise while every other method's grows. Past \(\sigma \approx 0.6\) — a rank correlation of 0.36 — a rule greedy on a noisy quantity does worse than a rule that ignores it.
What this takes away from the baselines chapter
Its figures — "0.0 to 1.0 % shortfall" — hold for known relevance. Under noise \(\sigma = 0.2\), the same filter leaves 5.0 %. On a real platform everything depends on the quality of its relevance estimator, which this repository has no way of measuring.
This does not change the substantive conclusion — what distinguishes this repository is not its algorithm — but it adds a third reason to think so: at realistic relevance, the gap between re-rankers matters far less than the quality of the underlying relevance engine.
3. What these two measurements change¶
The exchangeability test is confirmed, and that is the most useful result: it rejects under both click models, and more strongly than on real data. A log whose order says nothing says nothing under any model.
The severity estimator is restricted: it estimates an interpretable quantity only if examination does follow a power law — an assumption nothing in real data verifies, and one the literature explicitly doubts for results pages.
The baselines are restricted too: their ranking holds, their gap to chance does not.
Lead — now settled, without settling the question¶
~~Test the shape of examination, not only its existence.~~ → done. The third check exists: it never rejects under a position model and rejects massively under cascade. Applied to Baidu-ULTR it detects no cascade signature — but it meets a limit that is not its own: a click budget produces exactly the same signature as a cascade, and nothing in the clicks separates them. Settling it requires a measure of examination — display time, scroll depth — which Baidu-ULTR publishes and this repository has never read.
Implementation: ide.logs.simulate_cascade · Notebook:
21 — The two blind spots ·
counter-expertise · baselines ·
bibliography