The form test, and what it cannot separate¶
The third check exists and works
It never rejects under a position model (\(|z| < 1\) at every severity) and rejects massively under cascade (\(z = -97\) to \(-307\)). It answers the question the other two checks left open: in what form does order act?
But a click budget mimics a cascade
A reader who stops clicking once served — while still scanning the feed — produces the same signature: \(z = -144\) under a pure position model, against \(-179\) under a genuine cascade. The two are not separable from click data alone.
On Baidu-ULTR: no cascade signature
The standardised deviation is positive at every threshold — \(+5.6\) at the most permissive, \(+0.5\) to \(+0.6\) at the strictest — whereas a cascade would make it negative. The sign is that of the heterogeneity confounder, not of cascade.
And a protocol error, caught before publication
Restricting the log to multi-click sessions looked like the obvious way to separate budget from cascade. Applied to Baidu-ULTR it gives \(z = -8.2\) (\(p = 2 \times 10^{-16}\)) — a clear signature, which I was about to publish. On a pure position model, the same restriction moves \(z\) from \(+0.7\) to \(\mathbf{-45.7}\). It is a collider.
The construction¶
The two examination forms cannot be told apart by their shape: geometric and polynomial decay fit equally well over the first ranks. That is what made the problem look intractable.
They differ by a conditional independence:
- under a position model, examination of rank \(R\) depends only on \(R\). Given item and rank, the click is therefore independent of what happened above;
- under a cascade model, a click above suppresses examination below.
For each cell \(s = (\text{item}, \text{rank})\), the \(n_s\) impressions split into \(n_{1s}\) preceded by a click in the same feed and \(n_{0s}\) not, for \(k_s\) clicks in total. Under independence those clicks distribute as a draw without replacement, with exactly known moments:
This is the Mantel-Haenszel statistic, stratified by cell. Conditioning on the cell removes item quality — without it, feeds containing a click at the top are also those whose items are better, and the comparison would measure only that.
The two controls on pure models; the click budget that mimics cascade; the application to Baidu-ULTR; and the restriction that manufactures the signature it seeks. Figure regenerated by notebook 22.
The two controls¶
| Model | \(z\) | Cells | Verdict |
|---|---|---|---|
| position, \(\eta = 0.0\) | −0.33 | 2,197 | position-compatible |
| position, \(\eta = 1.0\) | +0.08 | 1,882 | position-compatible |
| position, \(\eta = 2.0\) | +0.92 | 952 | position-compatible |
| cascade, \(\gamma = 1.00\) | −307.4 | 2,032 | rejects |
| cascade, \(\gamma = 0.85\) | −178.7 | 1,660 | rejects |
| cascade, \(\gamma = 0.60\) | −96.7 | 1,010 | rejects |
The test therefore separates the two models perfectly when they are pure. That is the minimum condition, and it is met.
What it cannot separate¶
A reader who wants only one thing stops clicking once served, while still scanning the feed. Their examination is that of a position model; only click production stops. This is entirely ordinary behaviour.
| Click budget (position model) | \(z\) | Clicks per feed |
|---|---|---|
| unlimited | +0.74 | 0.990 |
| 3 | −2.15 | 0.979 |
| 2 | −21.1 | 0.917 |
| 1 | −144.2 | 0.661 |
| cascade \(\gamma = 0.85\), for reference | −178.7 | 0.750 |
Nothing in the clicks distinguishes them — as expected: in both cases the log shows exactly the same thing, namely that after a click there is nothing.
The difference nonetheless matters greatly for our purposes:
| After a click, the item below… | Consequence for a re-ranking | |
|---|---|---|
| cascade | is not examined | moving it up changes everything |
| click budget | is examined but will not be clicked | moving it up changes nothing |
The question this test was meant to settle — what exposure to attribute to deep ranks? — therefore remains open, and cannot be settled by clicks.
On real data¶
Baidu-ULTR is the repository's only log recording both served order and document identity: 64,200 sessions, 0.688 clicks per session, 53.4 % with no click at all.
| Impression threshold | \(z\) | \(p\) | Cells |
|---|---|---|---|
| 2 | +5.65 | \(1.6 \times 10^{-8}\) | 780 |
| 5 | +1.20 | 0.229 | 261 |
| 10 | +0.54 | 0.593 | 151 |
| 20 | +0.61 | 0.543 | 108 |
No cascade signature. The sign matters as much as the magnitude, and it is here that of the announced confounder: a reader more inclined to click clicks more everywhere, hence both above and below.
Three readings remain possible, and the repository cannot decide between them: examination there is close to a position model; or a cascade exists but the confounder masks it — the test is conservative, as it has always said; or coverage is too thin, 108 cells at the strictest threshold, because the same URL rarely reappears in a search log.
What this non-rejection does not do
It does not validate the power law. The blind spots showed it errs by a factor of 4,110 at rank twelve if examination is a cascade. Failing to detect a cascade with 108 cells does not establish its absence.
The error, and the check that caught it¶
53 % of Baidu-ULTR sessions have no click. The natural idea is to restrict the test to multi-click sessions — where a budget of one is excluded by construction, hence where cascade and budget should finally separate. Applied to Baidu-ULTR, the restriction gives \(z = -8.2\): a clear cascade signature on real data.
The same protocol, applied to logs whose truth is known:
| Model | \(z\) (whole log) | \(z\) (restricted) |
|---|---|---|
| position, unlimited budget | +0.74 | −45.7 |
| position, budget = 2 | −21.1 | −52.5 |
| cascade, \(\gamma = 0.85\) | −178.7 | no cells |
On a log simulated under a pure position model, with no cascade and no budget, the restriction manufactures a deviation of −45.7. The reason has a name: a feed's click count is a collider of its individual clicks, and conditioning on it induces negative dependence between them — this is Berkson's paradox.
The \(-8.2\) obtained on Baidu-ULTR therefore measures no cascade: it measures the selection I had just performed. The warning now sits in the function's documentation, where someone will have the idea of doing it again.
This is the third time in this repository that a seemingly reasonable protocol manufactures its own result, and the third time a check on simulated data catches it before publication. The rule that emerges deserves writing down: any protocol applied to real data must first be applied to data whose answer is known.
The three checks, now¶
| Check | Question | MIND | Baidu-ULTR |
|---|---|---|---|
| exchangeability | does the order say anything? | no (\(z = +0.12\)) | yes (\(z = -206\)) |
| identifiability | is there enough to estimate \(\eta\)? | yes, artificially | yes, 55 documents |
| form | does examination depend on upstream? | not applicable | not detected (\(z = +0.5\)) |
The third exists, works, and does not settle what we wanted settled.
Lead — settled, and it decided¶
~~Baidu-ULTR publishes displayed_time, serp_height and slipoff_count_after_click.~~ →
done. On the examination measure, cascade and budget separate
perfectly — \(z = -158\) to \(-412\) against \(-1.03\) — and cascade is refuted on Baidu-ULTR:
examination below a click is there more frequent, not less.
Better still: exposure stops being estimated. Measured directly, its severity is \(0.88 \pm 0.05\) over 143 documents, against \(1.09\) estimated over 55 from clicks — 23 % overstatement, because a click conflates examination with attractiveness.
Reproducibility¶
Baidu-ULTR's raw log is not versioned — 0.9 GB, CC BY-NC licence. The digest now carries the four counts per cell the test needs: impressions, impressions preceded by a click, clicks, and clicks preceded. No row designates a reader, and the result is recomputed identically — zero gap, which a test verifies.
This is also a useful property for the data access request: the form test is computable from aggregates, like the repository's other three measurements.
One exception, and what it signals
The restriction to multi-click sessions requires the full log — it cannot be computed from aggregate counts. That should have been a first warning: a protocol needing to return to individual rows performs a selection, and a selection has a price.
Implementation: ide.logs.upstream_dependence_test,
ide.logs.upstream_dependence_from_counts · Notebook:
22 — The form test ·
the two blind spots · MIND's exploration ·
logs that record the rank