Baselines, and the exact frontier that judges them¶
The repository's filter sits on the frontier
Median shortfall of 0.0 to 1.0 % of engagement depending on the floor, across 150 randomly drawn pools. Diversifying is easy; diversifying without leaving engagement on the table is not — random selection loses 9 to 19 % to reach the same levels.
But it is neither alone nor first
MMR, published in 1998, holds the frontier at every floor — 0.0 to 0.2 % — and even beats the filter head to head at floor 0.80 (35 wins to 18). This repository's novelty is not its algorithm.
It stands out where the requirement is high
At floor 0.90, MMR finds no compliant setting in 47 % of pools — its parameter saturates first — against 12 % for the entropic filter, which then wins 72 times to 14.
And the price of the standard depends on the reader
A floor of 0.90 costs 3.8 % of engagement when relevance is independent of viewpoint, and 17.1 % when it follows entirely from it. The standard costs most exactly where it serves most.
The question we had not asked¶
The filter proposed by this repository had only ever been compared with one opponent: ranking by pure relevance, from which it differs by construction. Nobody had asked whether it did better than a trivial heuristic.
It was the oldest debt of the evaluation programme, and the only one requiring no new data.
The protocol¶
A feed is judged on two quantities in tension: the engagement it produces and the diversity it exposes, both weighted by the attention of each rank. A re-ranker is therefore neither good nor bad: it occupies a point in the plane, and the only question worth asking is by how much it misses the attainable frontier.
That frontier is not estimated but computed exactly: all 151,200 ordered arrangements of six items drawn from a pool of ten are enumerated, and at each diversity level the maximum engagement is retained. It is an upper bound, not one more competitor.
| Method | Parameter swept |
|---|---|
| relevance ranking | — (reference) |
| round robin | number of head positions shared among viewpoints |
| MMR (Carbonell & Goldstein, 1998) | trade-off \(\lambda\) between relevance and spread |
| Boltzmann | temperature of the \(e^{r/T}\) draw |
| entropic filter (this repository) | coefficient \(\mu\) in \(\text{relevance} + \mu\,\Delta H\) |
| random | the seed — the baseline one forgets to plot |
Comparing methods where each happens to land would be unfair: their parameters do not correspond. So the floor is fixed and each is asked what it can do best while complying — exactly the regulator's question.
One pool, its exact frontier and what each method reaches on it; median shortfall across 150 pools; the share of pools where the floor is never reached; and the price of the standard by alignment between relevance and viewpoint. Figure regenerated by notebook 19.
What 150 pools give¶
Median shortfall, among pools where the method reaches the floor:
| Method | 0.50 | 0.60 | 0.70 | 0.80 | 0.90 |
|---|---|---|---|---|---|
| round robin | 0.0 % | 0.0 % | 0.0 % | 3.8 % | 4.8 % |
| MMR | 0.0 % | 0.0 % | 0.0 % | 0.0 % | 0.2 % |
| Boltzmann | 0.0 % | 0.0 % | 0.1 % | 1.2 % | 3.7 % |
| entropic filter | 0.0 % | 0.0 % | 0.0 % | 0.0 % | 1.0 % |
| random | 9.3 % | 9.8 % | 11.0 % | 12.9 % | 19.1 % |
Share of pools where the floor is never reached, though the frontier reaches it:
| Method | 0.50 | 0.60 | 0.70 | 0.80 | 0.90 |
|---|---|---|---|---|---|
| round robin | 0 % | 0 % | 0 % | 0 % | 28 % |
| MMR | 0 % | 0 % | 3 % | 0 % | 47 % |
| Boltzmann | 1 % | 1 % | 1 % | 8 % | 36 % |
| entropic filter | 0 % | 0 % | 1 % | 0 % | 12 % |
| random | 0 % | 0 % | 1 % | 0 % | 7 % |
The two tables must be read together, and it is their crossing that separates the methods. Random reaches floors more often than any other method — chance produces diversity — but pays 9 to 19 % for it. MMR pays nothing and fails half the time at the high requirement. The entropic filter is the only one doing both: reaching, and not paying.
The rule explains it: the filter optimises the constrained quantity directly, while MMR optimises a mean spread that is only a proxy — and a proxy that saturates.
The price of the standard depends on the reader¶
The comparison above uses pools where relevance is drawn independently of viewpoint. That is not neutral: it is the most favourable assumption for a diversity standard, since relevant items then exist in every viewpoint.
Varying the alignment — the share of relevance that follows from an item's viewpoint:
| Alignment | 0.60 | 0.70 | 0.80 | 0.90 |
|---|---|---|---|---|
| 0.00 — interests cut across viewpoints | 0.0 % | 0.3 % | 1.1 % | 3.8 % |
| 0.50 | 0.0 % | 0.8 % | 2.6 % | 5.6 % |
| 1.00 — the preference is a viewpoint | 2.2 % | 6.9 % | 8.6 % | 17.1 % |
The standard costs most exactly where it serves most. That is uncomfortable and must be said: it predicts where platforms' objection will fall — on the most polarised readers, the very case that motivates the standard.
It also reconciles two figures of this repository that seemed to contradict each other. Adversarial rank measured a cost of 10 to 21 % because relevance there was attached to the viewpoint — an alignment of 1. Here, at zero alignment, the same floor costs almost nothing. Not a contradiction: a dependence on the reader that neither had isolated.
What this comparison settles¶
The debt is paid. The filter has been judged against four competitors and against the exact bound, on 150 pools, at five levels of requirement.
The verdict is mixed, and that is the right result. The filter holds the frontier — which was not a given — but brings nothing a 1998 heuristic does not already bring, except at high floors. What distinguishes this repository is therefore not its algorithm: it is the standard and the instruments that verify it.
A consequence for the memorandum. Since a heuristic published in 1998 holds a floor of 0.80 with no measurable loss of engagement, the argument that a diversity standard would require redesigning recommender engines does not hold. What it requires is measuring it. → memorandum
Reservations¶
These pools are synthetic: ten items, four viewpoints, six positions. The exact frontier exists only at that scale — the price of exactness, the same as in adversarial rank. Nothing guarantees the relative ranking of methods transfers to a pool of a thousand items.
Relevance here is moreover known, whereas a platform has only its estimate. That caveat has since been measured: the ranking of methods holds at every noise level, but their advantage over random selection falls from 16.4 points at zero noise to 5.8 points at \(\sigma = 0.4\), and inverts beyond. Random is the only re-ranker that does not use relevance: its shortfall is invariant. → The two blind spots
Finally, random selection is treated generously here — eighteen draws per pool, keeping the best compliant one. A platform would not have that luxury, and its real performance would be worse than reported.
Implementation: ide.baselines · Notebook:
19 — Baselines ·
EDA — the algorithm · adversarial rank and severity ·
memorandum