Counter-expertise: what the literature holds against this work

A withdrawn conclusion: \"target proximity resists best\"

Published twice, it is a scale artefact. Comparing two diversity measures at the same nominal floor is meaningless — they do not live on the same scale. At equal actually exposed diversity, both measures cost the same, within 0.6 % of the exact bound.

A restricted conclusion: \"certified at 0.70, it exposes 0.36\"

That figure holds for a \(1/R\) attention discount, i.e. \(\eta = 1\). Yet this repository itself measured that \(\eta\) is a property of the surface: at \(\eta = 0.1\) — the three-thumbnail banner — burial disappears (0.747 exposed for 0.774 displayed); at \(\eta = 2\) it worsens (0.157). The discount must be measured, not conventional.

A widened uncertainty: trust bias

The repository's click model is multiplicative; the literature documents an affine one, where readers click out of trust on irrelevant top-ranked items. Under it, the estimator overstates severity by +12.8 % — and its standard error, 0.013, does not see it.

A confirmed recommendation: exploration cannot be substituted

The doubly robust estimator does worse than plain IPS on real data (−4.9 % against +2.5 %), and the effective sample size does not move: it depends only on the importance weights. No estimator manufactures exploration.

---

Why this page

Earlier chapters attacked the index, the standard, the datasets and the algorithm. They had never attacked the measuring instruments themselves, nor confronted the repository's choices with what the field already knows.

Five counter-tests, one of which invalidates a published conclusion.

Burial by attention severity; the price of both standards under the same severity; the two measures compared at equal exposed diversity; and the estimator's drift under trust bias. Figure regenerated by notebook 20.

1. The \(1/R\) discount is a convention, not a measurement

Severity \(\eta\)	Surface	Displayed	Exposed	Blind cost	Rank-aware cost
0.1	three-thumbnail banner	0.774	0.747	24.1 %	24.1 %
0.5	intermediate	0.774	0.620	17.7 %	21.1 %
1.0	MRR convention	0.774	0.443	10.7 %	20.9 %
1.1	search results page	0.774	0.408	9.5 %	20.9 %
2.0	very top-heavy feed	0.774	0.157	2.7 %	20.8 %

Burial exists only where attention is concentrated. With near-flat attention a rank-blind standard suffices, and the rank-aware one costs no more. With very concentrated attention, burying becomes nearly free — 2.7 % — hence irresistible.

The price of the rank-aware standard does not move: 20.8 to 24.1 % whatever the surface. It is the only stable quantity in the table.

Regulatory consequence. A regulator cannot fix a single discount. The attention discount is part of what must be measured on the surface, just as the served rank must be recorded. → Article 40 request

2. Comparing two measures at the same floor is meaningless

Adversarial rank compared four measures at the same 0.70 floor and concluded that target proximity resisted best. The honest comparison fixes the actually exposed diversity and asks each measure what it costs to deliver it.

Measure	Floor	Exposed entropy	Cost	Optimal cost
position entropy	0.60	0.605	16.4 %	16.4 %
target proximity	0.80	0.591	15.8 %	15.7 %
position entropy	0.70	0.702	20.9 %	20.8 %
target proximity	0.90	0.792	25.9 %	25.5 %

At comparable exposed diversity the costs are comparable too, and both measures sit on the exact bound. Target proximity does not resist better: it is less demanding at the same number.

What matters is therefore not the choice of measure but the level required and rank-awareness. The repository had applied the right method elsewhere — the baselines fix the floor precisely to make methods comparable — and had not applied it to its own measures.

3. Trust bias, which the estimator ignores

The unbiased learning-to-rank literature documents an affine rather than multiplicative model (Vardasbi, Oosterhuis & de Rijke, 2020):

\[P(\text{click} \mid k, \gamma) = \theta_k\big(\varepsilon^+_k\,\gamma + \varepsilon^-_k\,(1-\gamma)\big) = \alpha_k \gamma + \beta_k\]

The same authors prove that inverse propensity weighting cannot correct an affine bias: a linear transformation does not correct an affine one.

\(\varepsilon^-\)	\(\hat\eta\)	Standard error	Error
0.00	1.011	0.010	+1.1 %
0.05	1.045	0.010	+4.5 %
0.10	1.076	0.011	+7.6 %
0.20	1.128	0.013	+12.8 %

Two qualifications, in both directions: this is far less severe than positing \(\eta\) blindly, which costs up to +179 %; but the published interval is too narrow, and the real uncertainty includes a misspecification component the standard error does not estimate.

One technical detail explains why the estimator fares so well: trust bias independent of rank does not disturb it at all — the transformation stays separable and the item fixed effect absorbs it entirely. Only the rank dependence of trust biases it.

4. Doubly robust estimation does not save the effective size

Estimator	Value	Error
naive	0.006743	+31.6 %
IPS	0.005253	+2.5 %
self-normalised	0.004768	−7.0 %
IPS clipped at 100	0.005238	+2.2 %
doubly robust	0.004871	−4.9 %

Doubly robust does worse than plain IPS: its reward model, estimated on the logging bucket, inherits the very bias one set out to correct. And the effective sample size — 1,513 out of 4,077,727 — does not change: it depends only on the distribution of importance weights, that is, on the distance between the two policies.

The problem is structural. The memorandum's recommendation holds: require a fraction of exploration, not a better estimator.

5. Reading the index as an effective number of viewpoints

A normalised entropy is not a diversity: it is not linear in what we mean by "twice as diverse". Its conversion into an effective number is (Jost, 2006).

Index	Effective viewpoints (of 4)	Reading
0.774	2.92	displayed by the burying feed
0.700	2.64	proposed regulatory floor
0.443	1.85	actually exposed at \(\eta = 1\)
0.157	1.24	actually exposed at \(\eta = 2\)

"Certified at 2.6 effective viewpoints out of 4, it exposes 1.9" is understandable without training; "certified at 0.70, it exposes 0.44" is not.

6. What the literature adds, and measurement does not say

Correcting position bias does not necessarily improve ranking. Hager et al. (2024) revisit Baidu-ULTR — the very dataset where this repository measures \(\hat\eta = 1.10\) — and find that standard corrections improve click prediction without improving ranking quality as judged by expert annotators. They confirm the bias by four concordant methods, which corroborates our figure; but the next step does not follow mechanically.

A diversity standard is an editorial standard. The normative metrics of Vrijenhoek et al. (2022) split diversity into five dimensions derived from explicit democratic theories. This repository's index occupies one, and a platform can satisfy it by serving divergent, empty items. No automatic measure separates pluralism from false balance.

What the repository measures is exposure, not reception. The index bears on what is served, weighted by the presumed attention of a rank. What is actually read, understood or retained is not measured — and the attention discount is only a proxy for it, one that section 1 shows to be far from innocuous.

And our dataset conclusion is not new. van Drunen & Vrijenhoek (2025) establish, before this repository, that public datasets are the bottleneck for diversity-aware recommendation and that European law is the route to access. We corroborate; we do not discover.

→ Full bibliography

What this counter-expertise did not do — and which is now done

The two blind spots announced here have since been measured. → The two blind spots

• Under cascade, the exchangeability test holds: it rejects between \(z = -208\) and \(z = -241\), more strongly than on real data. The power law \(R^{-\eta}\), by contrast, collapses — at rank twelve it overstates real exposure by a factor of 40 to 4,110.
• Under estimated relevance, the ranking of methods holds but their advantage over random selection falls from 16.4 to 5.8 points, and inverts beyond.

---

Implementation: ide.radio.rank_weights (parameterised discount), ide.entropy.effective_viewpoints · Notebook: 20 — Counter-expertise · adversarial rank · baselines · bibliography