Compression as evaluation

Patrick Pynadath¹

¹ Purdue Department of Computer Science

Introduction

Motivation

Notation & evaluation setting

$\mathcal X$ : the space of finite text sequences. $P$ : the reference distribution. $Q$ : the generated distribution, including its decoding settings.

Symbols

Symbol	Meaning
$P,\ Q$	Reference and generated text distributions.
$x,\ y\in\mathcal X$	Individual text sequences.
$D_{\mathrm{fit}}$	Reference samples used to construct the dictionary.
$D_{\mathrm{test}}=(x_i)_{i=1}^{n}$	Held-out reference samples.
$G_{\mathrm{test}}=(y_j)_{j=1}^{m}$	Generated samples used for evaluation.
$G_{\mathrm{fit}}$	Separate generated samples for generated-context ablations.
$b(x)$	Length of the UTF-8 representation of $x$ , in bytes.
$T_\tau(x)$	Number of tokens under tokenizer $\tau$ .
$C$	A lossless compressor and its configuration.
$d_D=A(D_{\mathrm{fit}})$	Dictionary built by procedure $A$ from reference samples.
$L_C(x\mid d_D)$	Compressed length in bits, given the dictionary.
$L_C(x)$	Compressed length without an external dictionary.
$\delta_z$	A point mass at $z$ .
$f_\#P$	The distribution of $f(X)$ when $X\sim P$ .
$W_p(\mu,\nu)$	Order- $p$ Wasserstein distance.

For distributions on a Euclidean space,

W_p(\mu,\nu)= \left( \inf_{\pi\in\Pi(\mu,\nu)} \int \lVert u-v\rVert_2^p\,d\pi(u,v) \right)^{1/p},

where $\Pi(\mu,\nu)$ is the set of couplings with marginals $\mu$ and $\nu$ .

Information-theoretic quantities

H(P)=\mathbb E_{X\sim P}[-\log_2 P(X)], \qquad D_{\mathrm{KL}}(P\Vert Q) =\mathbb E_{X\sim P}\!\left[\log_2\frac{P(X)}{Q(X)}\right].

Units: bits. [1]

Coding rates & reference savings

r_{C,d_D}(x)=\frac{L_C(x\mid d_D)}{b(x)}.

r_C^{(0)}(x)=\frac{L_C(x)}{b(x)}, \qquad s_{C,d_D}(x)=r_C^{(0)}(x)-r_{C,d_D}(x).

Symbol	Meaning
$r_{C,d_D}(x)$	Reference-conditioned coding rate, in bits per UTF-8 byte.
$r_C^{(0)}(x)$	Standalone coding rate, in bits per UTF-8 byte.
$s_{C,d_D}(x)$	Reference savings, in bits per UTF-8 byte.

Existing metrics

Gradient Moment (GM)

Symbol	Meaning
$R_\theta$	Fixed pretrained evaluator.
$\mathcal L_\theta(x)$	Token-averaged negative log-likelihood of $x$ .
$g_\theta(x)$	Gradient of $\mathcal L_\theta(x)$ .

g_\theta(x)=\nabla_\theta\mathcal L_\theta(x).

\operatorname{GM}_\theta(P,Q) = \left\lVert \mathbb E_P[g_\theta(X)]-\mathbb E_Q[g_\theta(Y)] \right\rVert_2^2.

[2]

Energy distance

$\phi:\mathcal X\rightarrow\mathbb R^k$ : feature map. $X,X'\sim P$ and $Y,Y'\sim Q$ : independent samples.

\begin{aligned} \operatorname{ED}_{\phi}(P,Q) ={}&2\mathbb E\lVert\phi(X)-\phi(Y)\rVert_2\\ &-\mathbb E\lVert\phi(X)-\phi(X')\rVert_2\\ &-\mathbb E\lVert\phi(Y)-\phi(Y')\rVert_2, \end{aligned}

[2]

Sliced Optimal Transport for Language (SOL)

Symbol	Meaning
$h_t(x)\in\mathbb R^d$	Contextual token state at position $t$ .
$\mu_x$	Empirical token-state measure for $x$ .
$\mathsf P_h,\mathsf Q_h$	Distributions over document token-state measures.
$\xi$	Token-state projection direction.
$g$	Gaussian-process draw for the second projection.
$M_{\xi,g}$	Double-sliced projection of a document measure.

\mu_x=\frac{1}{T_\tau(x)} \sum_{t=1}^{T_\tau(x)}\delta_{h_t(x)}, \qquad h_t(x)\in\mathbb R^d.

\operatorname{SOL}(P,Q)^2 = d\,\mathbb E_{\xi,g} \left[ W_2^2\!\left( (M_{\xi,g})_\#\mathsf P_h,\, (M_{\xi,g})_\#\mathsf Q_h \right) \right].

[3]

MAUVE

$\widetilde P,\widetilde Q$ : quantized distributions of pretrained text embeddings. $M_\alpha$ : their mixture.

M_\alpha=\alpha\widetilde P+(1-\alpha)\widetilde Q, \qquad \alpha\in(0,1),

[4]

Desiderata for a language generation metric

Why compression

Distributions and prefix-free codes

$p$ : a discrete distribution. $\ell(x)$ : binary prefix-code length. $\ell^*(x)$ : optimum under the real-valued length relaxation.

\min_\ell \sum_x p(x)\ell(x) \quad\text{subject to}\quad \sum_x2^{-\ell(x)}\leq1.

\ell^*(x)=-\log_2p(x), \qquad \mathbb E_p[\ell^*(X)]=H(p).

[1]

Historical roots of compression in language modeling

Compression with modern LLMs

Quick primer on compression

[5]

Literal conversion

Huffman coding

Finite State Entropy coding

Our proposed compression metric

Definition

Definition: compare how well real and generated text compress using the same real-data dictionary.

\text{rate}(x)=\frac{\text{compressed bits of }x\text{ using the dictionary}}{\text{UTF-8 bytes in }x}

\boxed{\text{score}=W_1(\text{real-text rates},\;\text{generated-text rates})}

One rate per document: compress each document separately with the same dictionary; reset the compressor between documents.
Equal weight per document: compare the two lists of rates using one-dimensional Wasserstein distance.
Interpretation: lower means the rate distributions are closer; zero does not imply the text distributions are identical.
Units: bits per UTF-8 byte; no language-model evaluator required.

Computation

Algorithm

Build the dictionary from a reference-data fit split; keep the real evaluation documents separate.
Compress the real evaluation documents individually using that dictionary; record each document’s bits per byte.
Compress the generated documents in exactly the same way, using the same dictionary.
Compare the rate lists with Wasserstein distance.

For equal-sized lists: sort both lists, pair rates by rank, and average their absolute differences:

\text{score}=\frac{1}{n}\sum_{i=1}^{n}|a_i-b_i|,

Here: $a_i$ and $b_i$ are the sorted real and generated rates, respectively; each list contains $n$ documents.
For unequal-sized lists: use the standard one-dimensional Wasserstein calculation with equal weight per document within each list; do not truncate either list.
Our default: Zstandard level 10, window log 23, a 64 KiB raw real-data dictionary, and independent compression frames.

Experiments

Does it track downstream performance?

Question: does reference-conditioned compression distance track downstream task performance?
TL;DR: Compare compression and baseline metrics with published downstream accuracy across Pythia sizes, training checkpoints and decoding settings.
Core results: The association depends on decoding; use the correlation tables and within-size checks below rather than treating the pooled trend as universal.

Experimental setup, methodology & interpretation

Metric: forward conditional-rate Wasserstein-1; smaller distances indicate closer coding-rate distributions.
Comparators: SOL, handcrafted-feature energy distance, Gradient Moment, and Qwen3-8B-Base negative log-likelihood; all displayed scores use lower-is-better orientation.
Scope: Pythia; separate analyses for each decoding setting.

Experimental setup

Model sizes: 160M, 410M, 1B, and 6.9B.
Training checkpoints: 3K, 13K, 33K, 63K, 103K, and 143K updates per size.
Decoding: temperatures 0.7, 1.0, and 1.3 without truncation; temperature 1.0 with top-p 0.95; temperature 1.0 with top-k 20.
Generation pool: 1,024 unconditional sequences per setting; 1,024 model-native tokens per sequence; generation seed 2026; EOS/BOS prompt excluded from evaluated text; generation continues through EOS.
Displayed analysis: cached 512-sequence candidate bank split into 256 fit and 256 held-out sequences; 512 Pile reference sequences split into 256 fit and 256 held-out sequences; split seed 2026.
Forward score: real-data fit dictionary only; 256 held-out real sequences versus 256 held-out generated sequences; generated fit samples are unused by this direction.
Downstream target: unweighted mean accuracy across seven published five-shot evaluations: ARC-Challenge, ARC-Easy, LAMBADA, LogiQA, PIQA, SciQ, and WinoGrande; WSC and MMLU excluded from this aggregate.
Observation count: 24 size–checkpoint pairs per decoding setting; 120 metric measurements overall; five decoding settings share each checkpoint’s downstream target.

Measurement and analysis

Compressor: Zstandard level 10; window log 23; fixed 64 KiB raw reference dictionary.
Coding protocol: independent compression frame per sequence; adaptation within each sequence permitted; no context carried between evaluated sequences.
Normalization: compressed bits per original UTF-8 byte; frame overhead included; shared dictionary transmission excluded.
Comparison: exact empirical one-dimensional Wasserstein-1 between conditional coding rates; equal sequence weights; no histogram bins.
Association: Spearman correlation between negative distance and downstream mean accuracy; positive correlation means lower distance accompanies higher accuracy.
Comparator sample counts: SOL, energy distance, and Gradient Moment use 512 generated sequences per setting; Qwen likelihood uses 1,024; conditional compression uses 256 held-out sequences. Same 24 checkpoints, different cached sample budgets; not a sample-matched head-to-head comparison.
Comparator inputs: SOL and Gradient Moment retokenize the text with a 1,024-token cap; energy distance uses the full text; sequence counts and tokenizer horizons differ from paper-default protocols.
Likelihood: token-weighted Qwen3-8B-Base negative log-likelihood, expressed in bits per Pythia token; no compression penalty added.
Unigram entropy: no complete saved Pythia checkpoint table found; omitted rather than replaced with compression rate or combined with likelihood using an arbitrary sum.
Controls: separate pooled correlations, within-size correlations, and leave-one-size-out correlations; no fitted regression line or independence-based significance test.

Reading the figures

Scatter: one point per size–checkpoint pair; color and marker shape identify model size; horizontal axis is the selected evaluation score, lower is better; vertical axis is downstream accuracy, higher is better; hover or focus a point for its update and values.
Training curves: separate distance and downstream panels; shared update axis; fixed axes across decoding choices; no dual-axis overlay; lines connect observed checkpoints only.
Correlation table: all five decoding settings, including T=1.3; pooled correlations use 24 points; within-size correlations use six; leave-one-size-out correlations use 18.
Metric comparison table: pooled correlations for every metric and decoding setting; correlations share a scale, raw metric values do not.

Limitations

Dependence: checkpoints share training trajectories; decoding variants share downstream measurements; 120 settings are not 120 independent observations.
Confounding: pooled correlation can reflect model size rather than within-size training improvement; six checkpoints per size support descriptive trends, not precise within-size inference.
Sampling: one generation seed and one displayed reference split; no uncertainty intervals shown; packed reference rows can share original source documents.
Interpretation: association with this seven-task aggregate, not a causal effect or a validated general measure of language quality.
Sources: published Pythia evaluation archive; local conditional-direction analysis 8630a5e84115aef93a3a; exact source paths and SHA-256 hashes in the downloadable data.

Loading saved results…

All 24 plotted points · selected decoding setting

Download all 120 measurements and source hashes ↗

Does it detect adversarial samples?

Question: do artificial samples receive worse scores than real OpenWebText, and by how much?
TL;DR: Score four families of synthetic text against a real OpenWebText control.
Core results: Conditional compression, SOL, energy distance, Gradient Moment and JCD place all 73/73 settings above the real control; Qwen NLL does so for 40/73. These are observed margins, not significance tests.

Experimental setup, methodology & interpretation

Scope: four França-inspired sampler families; 73 parameter/seed configurations; real-data control from the same experiment suite [2].

Samplers

Sampler	Construction in this experiment	Configurations
Mirror	Sample an IID token half-sequence, then append an exact copy; not a reversal.	18
Periodic	Repeat a fixed cycle of frequent tokens; identical sequences across the sampled collection.	10
Phrase bank	Concatenate uniformly sampled common five-token phrases.	27
Top-k IID	Independently draw from frequent tokens, weighted by corpus frequency.	18

Experimental setup

Source: OpenWebText construction samples supply token frequencies and phrase banks; synthetic sequences contain 1,024 GPT-2 tokens.
Sweep: vocabulary or phrase-bank size and generation seed; configurations are related settings, not 73 independent datasets.
Conditional compression: Zstandard level 10, window log 23, fixed 64 KiB raw real-data dictionary; 256 held-out real and 256 held-out generated sequences; independent frames; equal sequence weights.
Likelihood: Qwen3-8B-Base negative log-likelihood on 512 sequences per configuration, including the real control; token-weighted bits per GPT-2 source token.
Baselines: SOL, energy distance, Gradient Moment, and JCD use 512-sequence input banks and their cached metric-specific protocols; JCD includes internal fit/test splits.
Alignment: same sampler configurations across metrics; sample budgets and internal splits differ; not a sample-matched head-to-head comparison.

Margin above the real-data control

\Delta M_a=M(G_a,D)-M(D_{\mathrm{control}},D).

Notation: $G_a$ is the sample bank for adversarial configuration $a$ ; $D$ is the real reference bank; likelihood uses the fixed Qwen evaluator rather than a fitted reference-bank model.
Positive margin: the adversary receives a worse score than the real control.
Negative margin: the adversary receives a better score than the real control.
Zero line: equality with the observed real control; not a statistical decision threshold.
Dots: individual configurations; vertical offsets separate overlapping points and have no statistical meaning.
Short vertical marks: family medians; computed from unrounded scores.
Axes: independent linear scales in each metric’s native units; raw margin magnitudes are not comparable across metrics.

Interpretation limits

Control uncertainty: one real-control configuration in this sweep; no confidence bands or significance claims.
Ratios: not used; near-zero or negative real-control scores can make ratios unstable or misleading.
Detection counts: number of configurations with strictly positive observed margins; not independent trials or a general detection rate.
Scope: paired structural corruptions and compression-matched attacks belong to separate experiments; not pooled into this result.

Loading saved adversarial results…

● Positive margin● Negative margin│ MedianDashed line: real control

Hover, focus, or tap a point for its parameters and scores.

All 73 configurations · raw scores and margins

Download scores, unrounded margins, and source hashes ↗

Corruption & collapse sweeps

Question: how do scores change as corruption or repetition increases?
TL;DR: Increase token corruption, repetition and document collapse, then compare each metric with its real-data control.
Core results: Conditional compression places 35/39 settings above the real control; Qwen NLL 25/39, SOL 39/39, energy distance 39/39, Gradient Moment 38/39 and JCD 32/39. Counts summarize observed margins, not statistical significance.

Experimental setup, methodology & interpretation

Coverage: 39 configurations; six metrics; the same real-data control as the França sampler sweep.
Token pollution: replace selected tokens with the token for “ the”.
Token shuffle: permute selected token positions; includes whole-document and localized regions.
Repetition: replace a suffix with repetitions of the first 16 tokens.
Collapse: replace a fraction of documents with one shared document.
Finite bank: replace documents with draws from a small shared bank.
Budgets: conditional W₁ uses 256 held-out sequences per bank; Qwen NLL scores 512 sequences; SOL, energy distance, Gradient Moment and JCD use 512-sequence input banks with their own internal protocols.
Interpretation: severity ranges from 0.01 to 1 for the main sweeps; localized shuffling and pollution use severity 1 within a selected fifth of the document. Inspect severity and region rather than treating every setting as a separate successful detection.
Reading the plot: positive margins indicate higher scores; negative margins indicate lower scores; each metric retains its native units. Hover or focus on a point for exact setting details.
Uncertainty: observed differences, not significance tests; settings and seeds are related, not independent detection trials.
Data: download all five suites, raw scores, controls and source hashes.

Loading saved results…

Adaptive adversaries

Question: can samples preserve local patterns or match aggregate compressibility while changing the distribution?
TL;DR: Construct collages, noise/repetition mixtures and restricted document banks to probe what the scores miss.
Core results: On the 14 held-out settings, conditional compression and all four distributional baselines exceed their real-control means in 14/14 cases; Qwen NLL does so in 10/14. These are descriptive comparisons.

Experimental setup, methodology & interpretation

Coverage: 14 settings in each of development and held-out phases; three independent real-control subsets per phase.
Collage: concatenate real-text spans of 8, 32, 128 or 512 tokens from randomly selected source documents.
Mixture: an IID token prefix followed by a repeated phrase; vary the noise fraction from 0 to 1, including intermediate compression-matching candidates.
Narrow topic: resample a 128-document bank selected using computing, science or politics keywords.
Random bank: resample a random 128-document bank; separates finite support from topic selection.
Budgets: conditional W₁ uses 128 held-out sequences per bank; Qwen and baseline metrics use 256-sequence input banks.
Controls: subtract the mean of the three real-control scores from the same phase; tables retain their minimum and maximum. These ranges are not confidence intervals.
Selection: held-out results shown first; development remains separately selectable, never pooled with held-out results.
Reading the plot: positive margins indicate higher scores; negative margins indicate lower scores; each metric retains its native units. Hover or focus on a point for exact setting details.
Uncertainty: observed differences, not significance tests; settings and seeds are related, not independent detection trials.
Data: download all five suites, raw scores, controls and source hashes.

Loading saved results…

Structural adversaries

Question: which changes to support and structure remain visible to each metric?
TL;DR: Change document support, shuffle sentence chunks or replace digits, and compare with independent real subsets.
Core results: Conditional compression exceeds the control mean in 8/13 held-out settings; Qwen NLL, SOL and energy distance in 9/13, Gradient Moment in 12/13 and JCD in 8/13. Not every setting is necessarily degraded text; the separate paired tests isolate transformation effects.

Experimental setup, methodology & interpretation

Coverage: 13 settings per phase; three real-control subsets per phase; development and held-out phases kept separate.
Unique topic documents: keyword-selected computing, science or politics documents without repeated-text sampling.
Random unique documents: a support-selection control, not necessarily degraded language.
Disjoint collage: real spans assembled using disjoint source pools for generated reference and test banks.
Sentence shuffle: permute selected punctuation-delimited chunks at fractions 0.25, 0.5 and 1.0; heuristic boundaries, not a linguistic sentence parser.
Digit substitution: change selected digits with a nonzero modular shift at fractions 0.25 and 1.0; grammaticality need not change.
Budgets: conditional W₁ uses 128 held-out sequences per bank; Qwen and baseline metrics use 256-sequence input banks.
Controls: phase-specific mean of three real subsets; table includes the observed control range.
Important distinction: these are comparisons with independent real subsets, not same-document before/after comparisons. Use the paired tests below to isolate transformation effects.
Reading the plot: positive margins indicate higher scores; negative margins indicate lower scores; each metric retains its native units. Hover or focus on a point for exact setting details.
Uncertainty: observed differences, not significance tests; settings and seeds are related, not independent detection trials.
Data: download all five suites, raw scores, controls and source hashes.

Loading saved results…

Paired structural tests

Question: does transforming the same source documents increase the score?
TL;DR: Score the same source documents before and after sentence shuffling or digit substitution.
Core results: Blind spot: the current real-context conditional W₁ decreases for all five transformations instead of penalizing them. This panel reports the original rate-W₁ variants, not the newer dictionary-difference candidates.

Experimental setup, methodology & interpretation

Pairing: original and transformed versions of the same source banks; fixed real context and real evaluation bank; generated context and generated test banks transformed together.
Conditions: sentence shuffle at 0.25, 0.5 and 1.0; digit substitution at 0.25 and 1.0.
Budget: 256 sequences in each real/generated fit/test bank; fixed 64 KiB raw dictionaries; Zstandard level 10, window log 23.
Metric variants: real-context conditional W₁, generated-context conditional W₁, their sum, and standalone rate W₁; all measured in bits per UTF-8 byte.
Margin: transformed score minus original score, not minus an independent real control.
Blind spot: real-context conditional W₁ decreases in all five conditions; it does not penalize these transformations in this paired experiment.
Meaning: shuffled chunks can retain many reusable substrings; digit changes can preserve local coding statistics. This is limited sensitivity to these transformations, not proof of semantic equivalence.
Coverage limit: this paired snapshot contains compression metrics only; unpaired Qwen, SOL, energy-distance, GM and JCD scores are not substituted for missing paired measurements.
Reading the plot: positive margins indicate higher scores; negative margins indicate lower scores; each metric retains its native units. Hover or focus on a point for exact setting details.
Uncertainty: observed differences, not significance tests; settings and seeds are related, not independent detection trials.
Data: download all five suites, raw scores, controls and source hashes.

Loading saved results…

Standalone compression-matched adversaries

Question: can matching standalone compression rates evade the conditional metric?
TL;DR: Select synthetic samples to match real standalone compression rates, then evaluate on held-out real data.
Core results: Conditional W₁ exceeds its real control in 3/3 constructions; standalone W₁ in 2/3. This is evidence against this particular standalone-matching attack, not a significance test.

Experimental setup, methodology & interpretation

Candidates: 3,840 phrase-bank and top-k IID samples per construction seed.
Selection: greedily choose 128 samples without replacement to match 128 real tuning-sample standalone rates; no conditional score used in selection.
Evaluation: independent 128-sample real holdout and 128-sample real control; fixed 64 KiB real dictionary from the original 256-sample fit bank; Zstandard level 10, window log 23.
Repetitions: three seeds vary candidate samples and the real tuning/evaluation split; not three independent datasets.
Comparison: standalone rate W₁ versus real-context conditional rate W₁, using the same selected samples and held-out real bank.
Result: conditional W₁ exceeds its matched real control in all three constructions; standalone W₁ exceeds its control in two of three.
Scope: this attack targets standalone rate distributions, not the conditional metric itself; no paired likelihood or learned-baseline scores are available in this snapshot.
Reading the plot: positive margins indicate higher scores; negative margins indicate lower scores; each metric retains its native units. Hover or focus on a point for exact setting details.
Uncertainty: observed differences, not significance tests; settings and seeds are related, not independent detection trials.
Data: download all five suites, raw scores, controls and source hashes.

Loading saved results…

Interpretability

Alternative metrics we tried

Selection criterion: evaluate against reference data without requiring a pretrained language-model evaluator.
Reading this section: distinguish observed failures, incomplete diagnostics, and methods excluded because they do not meet the model-free objective.
Not a claim that every earlier metric failed: several detect every França setting; those successes alone do not establish sensitivity to structure or distributional similarity.

The metric we decided on

Current main candidate: Wasserstein distance between per-document coding rates under the same fixed real-data context.
Why retain it: entirely compressor-based; references determine the context; responds to all three standalone-rate-matched attacks in the separate held-out test.
Why keep ablations: standalone rates, generated-context rates and the bidirectional sum expose which parts of the score contribute.
Known limit: real-context conditional W₁ decreases for all five paired structural transformations; the bidirectional sum does not fix that result.
Evidence: standalone-matched attacks, paired structural tests, downstream evaluation.

Earlier candidates: evidence & trade-offs

How to read this comparison: select a candidate to see its motivation, construction, observed results, and assessment. A method can pass a sampler test yet miss our model-free or structural-sensitivity criteria.

Shared notation & comparison rules

Notation: $h_R(Q)$ is generated-text cross-entropy under the Qwen evaluator $R$ ; $c(Q)$ is the standalone collection compression rate; $g(Q,P)=|c(Q)-c(P)|$ ; $d_C(Q,P)$ is a reference-versus-own-context coding difference. Rates use the same token normalization within each comparison.
Criteria: model-free operation, reference sensitivity, adversarial separation, structural sensitivity and useful downstream association.
Counts: strictly above a real control, or increases in a paired test; not significance tests. Protocols and budgets differ across experiments; compare within each experiment.

Likelihood alone

Why this was reasonable

A reference language model assigns code lengths to text; fluent, familiar sequences should generally receive shorter codes.

Definition & computation

Construction: $h_R(Q)$

Observed results

Variant	Real control	Mirror	Periodic	Phrase bank	Top-k IID
Qwen NLL	3.71061	9/18	1/10	12/27	18/18

Protocol: 512 matched test sequences; bits per GPT-2 source token; compression averages three corpus-order seeds. Counts are settings above the observed real control.

Downstream association

Decoding	Spearman ρ
t07	-0.914
t10	0.942
t10-k20	0.051
t10-p095	0.937
t13	-0.334

Protocol: 24 Pythia size/checkpoint points per setting; correlation of negative score with the seven-task five-shot average; historical 512-sequence scoring subset. Shared training trajectories are not independent trials.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: França sweep: many mirror, periodic and phrase-bank settings score below real data.
Decision: Keep as learned baseline, not the model-free metric.

“Entropy gap” alone

Why this was reasonable

Matching information content should expose both overly repetitive output and excessively random output.

Definition & computation

Construction: $g(Q,P)$

Observed results

Variant	Real control	Mirror	Periodic	Phrase bank	Top-k IID
Absolute compression gap	0.03131	18/18	10/10	27/27	18/18

Protocol: 512 matched test sequences; bits per GPT-2 source token; compression averages three corpus-order seeds. Counts are settings above the observed real control.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: Detects all four França families in the saved sweep; matching one average rate still cannot establish distributional similarity. The separate standalone-W₁ attack targets a stronger rate-distribution summary, not this exact collection-gap formula.
Decision: Useful diagnostic, not sufficient by itself.

Likelihood + gap

Why this was reasonable

Likelihood tests plausibility while a compression-gap penalty discourages both collapse and excess randomness.

Definition & computation

Construction: $h_R(Q)+g(Q,P)$

Observed results

Variant	Real control	Mirror	Periodic	Phrase bank	Top-k IID
NLL + gap	3.74192	18/18	10/10	27/27	18/18

Protocol: 512 matched test sequences; bits per GPT-2 source token; compression averages three corpus-order seeds. Counts are settings above the observed real control.

Downstream association

Decoding	Spearman ρ
t07	0.970
t10	0.869
t10-k20	0.973
t10-p095	0.810
t13	-0.383

Protocol: 24 Pythia size/checkpoint points per setting; correlation of negative score with the seven-task five-shot average; historical 512-sequence scoring subset. Shared training trajectories are not independent trials.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: Detects all four França families; still requires Qwen.
Decision: Earlier successful hybrid; excluded from the model-free main metric, not declared universally unsuccessful.

VCDIFF reference difference

Why this was reasonable

Under ideal distribution-matched coding, the real-versus-own coding overhead on generated samples has a KL interpretation.

Definition & computation

Construction: $d_C(Q,P)$

Observed results

Variant	Real control	Mirror	Periodic	Phrase bank	Top-k IID
VCDIFF coding difference	0.00656	0/18	10/10	27/27	0/18

Protocol: 512 matched test sequences; bits per GPT-2 source token; compression averages three corpus-order seeds. Counts are settings above the observed real control.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: Sweep: no mirror or top-k IID setting exceeds the real control; pilot forward scores are exactly zero for both.
Decision: Context matching alone leaves blind spots.

Reverse VCDIFF difference

Why this was reasonable

Evaluating real samples under a generated context tests whether generated data covers the reference distribution.

Definition & computation

Construction: $d_C(P,Q)$

Observed results

Sample	Reverse coding difference
Real OWT	0.005384
GPT-2	0.043419
Periodic	0.062174
Mirror	0.062174
Phrase bank	0.062174
Top-k IID	0.062174
Collapsed documents	0.062174

Protocol: separate pilot; bits per source token; three reference-order seeds, not independent datasets. Do not compare raw scores with the larger sweep.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: Pilot: identical mean reverse scores for periodic, mirror, phrase-bank and top-k IID samples.
Decision: Reversing direction alone does not provide useful separation among these cases.

VCDIFF + entropy gap

Why this was reasonable

An information-content penalty could catch degeneracies that reference matching alone overlooks.

Definition & computation

Construction: $d_C(Q,P)+g(Q,P)$

Observed results

Sample	Forward difference + entropy gap
Real OWT	0.044830
GPT-2	2.014083
Top-k IID	4.200320
Collapsed documents	49.196604
Periodic	41.594454
Phrase bank	11.640584
Mirror	7.954758

Protocol: separate pilot; bits per source token; three reference-order seeds, not independent datasets. Do not compare raw scores with the larger sweep.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: Pilot: the gap supplies a positive penalty for mirror and top-k IID when their forward VCDIFF difference is zero; exact pilot scores follow.
Decision: Model-free hybrid, but still inherits the information loss of a scalar rate gap.

Zstd concatenation difference

Why this was reasonable

Compressing a reference prefix followed by test text offers a simple off-the-shelf approximation to contextual coding.

Definition & computation

Construction: Difference of incremental code lengths after real versus generated prefixes

Observed results

Variant	Real control	Mirror	Periodic	Phrase bank	Top-k IID
Concatenation coding difference	-0.00382	18/18	10/10	27/27	18/18

Protocol: 512 matched test sequences; bits per GPT-2 source token; compression averages three corpus-order seeds. Counts are settings above the observed real control.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: All four França families exceed the control in the larger sweep; the initial periodic concern is not a failure under this strict observed-margin criterion. Adaptive target coding can dilute prefix effects.
Decision: Retain as historical comparator; not a frozen-context cross-entropy estimator.

Likelihood + maximum penalty

Why this was reasonable

The larger of the contextual and information-content penalties could cover the weaknesses of either alone.

Definition & computation

Construction: $h_R(Q)+\max\{d_C(Q,P),g(Q,P)\}$

Observed results

Variant	Real control	Mirror	Periodic	Phrase bank	Top-k IID
NLL + max(VCDIFF, gap)	3.74192	18/18	10/10	27/27	18/18
NLL + max(concatenation, gap)	3.74192	18/18	10/10	27/27	18/18

Protocol: 512 matched test sequences; bits per GPT-2 source token; compression averages three corpus-order seeds. Counts are settings above the observed real control.

Downstream association

Decoding	Spearman ρ
t07	0.970
t10	0.869
t10-k20	0.973
t10-p095	0.810
t13	-0.383

Protocol: 24 Pythia size/checkpoint points per setting; correlation of negative score with the seven-task five-shot average; historical 512-sequence scoring subset. Shared training trajectories are not independent trials.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: All four França families exceed the control for both VCDIFF and concatenation variants.
Decision: More complex and still model-dependent; no need to call this a failed detector.

Standalone rate W₁

Why this was reasonable

A distribution of per-document rates preserves variation that an average compression rate discards.

Definition & computation

Construction: Compare distributions of $r_C^{(0)}(x)$

Observed results

Seed	Standalone W₁	Real control	Conditional W₁	Conditional control
0	0.06184	0.05166	0.29891	0.03569
1	0.03719	0.03542	0.25992	0.03037
2	0.02304	0.03173	0.25463	0.02493

Protocol: 128 selected synthetic samples; independent 128-sample real evaluation and control banks; selection uses only standalone rates. Bits per UTF-8 byte. This attack targets standalone W₁, not the conditional metric.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: Three specially selected attacks closely match standalone rates; conditioning on real context increases separation in all three.
Decision: Useful ablation; does not encode reference-pattern overlap.

Bidirectional conditional W₁

Why this was reasonable

Using both real and generated contexts might capture complementary differences missed by either direction.

Definition & computation

Construction: Add real-context and generated-context W₁

Observed results

Paired transformation	Before	After	Change
digit_substitution-0.25	0.052287	0.042538	-0.009749
digit_substitution-1.0	0.052287	0.049127	-0.003161
sentence_shuffle-0.25	0.052287	0.045995	-0.006292
sentence_shuffle-0.5	0.052287	0.044316	-0.007972
sentence_shuffle-1.0	0.052287	0.046118	-0.006169

Protocol: exact original/transformed source banks; 256 documents per fit/test bank; bits per UTF-8 byte. All five changes are negative.
Sources: historical results, paired and matched diagnostics; detailed protocols appear in the corresponding experiment sections.

Criteria & decision

Evidence / limitation: Paired structural test: the sum decreases in all five conditions.
Decision: Additional diagnostic, not a cure for structural insensitivity.

Zstd dictionary difference · mean over generated documents

Why this was reasonable

Use the same Zstd dictionary protocol as the current metric, but directly measure the extra cost of coding generated documents with real rather than generated context.

Definition & computation

Per-document quantity:

\Delta(x)=\frac{L_C(x\mid d_D)-L_C(x\mid d_G)}{b(x)}.

Score:

S_{\mathrm{mean}}=\frac1m\sum_{j=1}^m\Delta(y_j).

Construction: Average the signed difference over generated test documents only; retain negative values.
Protocol: current Zstandard level 10, window log 23; equal raw-dictionary byte budgets capped at 64 KiB; separate fit/test banks; independent document frames; bits per UTF-8 byte. Reuses cached rates across 1,176 configurations; no new generation or compression.

Observed results

França samplers: 73/73 settings above the real control, including mirror and top-k IID.
Corruption sweeps: repetition 0/7; pollution 6/10; shuffling 8/10; collapse 4/7 settings above the real control.
Paired structural tests: 5/5 transformations increase the score relative to the same original documents; not a significance claim.
Downstream: Spearman correlation of negative score with the seven-task five-shot downstream average, across 24 Pythia size/checkpoint points per setting: t07: -0.645; t10: 0.469; t10-k20: -0.188; t10-p095: 0.410; t13: 0.261.
Source: download the analyses, paired scores, protocol descriptions and source hashes.

Criteria & decision

Interpretation / decision: A KL-motivated ablation, not an exact KL estimate. Detects all França settings but misses the repetition sweep; downstream correlations depend on decoding.
Limits: inherited source-overlap caveats remain; budgets vary by cohort; repeated checkpoints are not independent model runs.

Zstd dictionary difference · Wasserstein over documents

Why this was reasonable

Keep a signed context-preference score for every document, then compare its distribution on real and generated text; this avoids reducing generated samples to a single mean.

Definition & computation

Per-document quantity:

\Delta(x)=\frac{L_C(x\mid d_D)-L_C(x\mid d_G)}{b(x)}.

Score:

S_{\mathrm{contrast}}=W_1\big(\{\Delta(x_i)\}_{i=1}^n,\{\Delta(y_j)\}_{j=1}^m\big).

Construction: Compute the same signed difference for real and generated test documents, then compare their distributions with equal document weights. Do not reverse the subtraction on real documents or take absolute values before transport.
Protocol: current Zstandard level 10, window log 23; equal raw-dictionary byte budgets capped at 64 KiB; separate fit/test banks; independent document frames; bits per UTF-8 byte. Reuses cached rates across 1,176 configurations; no new generation or compression.

Observed results

França samplers: 73/73 settings above the real control, including mirror and top-k IID.
Corruption sweeps: repetition 7/7; pollution 10/10; shuffling 10/10; collapse 6/7 settings above the real control.
Paired structural tests: 2/5 transformations increase the score relative to the same original documents; not a significance claim.
Downstream: Spearman correlation of negative score with the seven-task five-shot downstream average, across 24 Pythia size/checkpoint points per setting: t07: 0.635; t10: 0.688; t10-k20: 0.729; t10-p095: 0.454; t13: 0.097.
Source: download the analyses, paired scores, protocol descriptions and source hashes.

Criteria & decision

Interpretation / decision: Promising alternative, not a rejected version: stronger corruption-sweep separation than the generated-only mean, but mixed paired structural sensitivity. Retain as a candidate; the main definition remains unchanged.
Limits: inherited source-overlap caveats remain; budgets vary by cohort; repeated checkpoints are not independent model runs.

Earlier metrics: quantitative comparisons

Comparison below: 512 matched test sequences per setting; bits per GPT-2 source token; means over three corpus-order seeds for compression; Qwen3-8B-Base likelihood. Counts mean strictly above the observed real-control score, not statistical significance.

Candidate	Real control	Mirror	Periodic	Phrase bank	Top-k IID
Qwen NLL	3.71061	9/18	1/10	12/27	18/18
Absolute compression gap	0.03131	18/18	10/10	27/27	18/18
NLL + gap	3.74192	18/18	10/10	27/27	18/18
VCDIFF coding difference	0.00656	0/18	10/10	27/27	0/18
Concatenation coding difference	-0.00382	18/18	10/10	27/27	18/18
NLL + max(VCDIFF, gap)	3.74192	18/18	10/10	27/27	18/18
NLL + max(concatenation, gap)	3.74192	18/18	10/10	27/27	18/18

Historical VCDIFF pilot: separate sample banks and protocol; do not pool its scores with the sweep above. Mirror and top-k IID both had an exactly zero forward coding difference; reversing the direction gave the same mean for several very different samplers.

Pilot sample	Forward VCDIFF	Reverse VCDIFF	Forward + entropy gap
Real OWT	-0.005384	0.005384	0.044830
GPT-2	0.037907	0.043419	2.014083
Periodic	29.421844	0.062174	41.594454
Mirror	0.000000	0.062174	7.954758
Phrase bank	2.361336	0.062174	11.640584
Top-k IID	0.000000	0.062174	4.200320
Collapsed documents	37.038979	0.062174	49.196604

Units: pilot scores are bits per source token; values averaged over the three stored reference-order seeds, not three independently generated datasets.

Downstream correlation depends on decoding

Pythia: 24 size/checkpoint points per decoding setting; seven-task five-shot downstream average; Spearman correlation with the negative metric, so positive means lower scores align with higher downstream accuracy. Repeated checkpoints are not independent model runs.
Protocol: historical 512-sequence scoring subset; not the larger likelihood budget used in the main downstream section.

Decoding	NLL	NLL + gap	NLL + max(VCDIFF, gap)	NLL + max(concatenation, gap)
t07	-0.9139	0.9704	0.9704	0.9704
t10	0.9417	0.8687	0.8687	0.8687
t10-k20	0.0513	0.9730	0.9730	0.9730
t10-p095	0.9374	0.8104	0.8104	0.8104
t13	-0.3339	-0.3826	-0.3826	-0.3826

Takeaway: adding the gap improves some decoding settings and worsens others; the maximum-penalty variants have identical rank correlations here, not evidence of a universal improvement.

The obvious baseline: likelihood minus entropy

Yes, the natural theoretical baseline is cross-entropy minus generated-distribution entropy:

H(Q,R)-H(Q)=D_{\mathrm{KL}}(Q\Vert R), \qquad H(Q,R)=\mathbb E_{X\sim Q}[-\log_2 R(X)].

Target distinction: this is KL to the evaluator distribution $R$ . It equals KL to the real-data distribution $P$ only if $R=P$ .
Practical baseline: replace $H(Q)$ with a Zstandard coding rate, using the same source-token denominator as the Qwen NLL:

J_C(Q;R)=h_R(Q)-c(Q).

Approximation caveat: Zstandard’s finite-sample coding rate is not the true entropy rate; Qwen is not the true reference distribution. This subtraction is a diagnostic residual, without KL’s guaranteed nonnegativity or identity-of-indiscernibles.
Reference-centered version: compare that residual with the measured real-control residual:

\Delta J_C(Q,P;R) =\big[h_R(Q)-c(Q)\big]-\big[h_R(P)-c(P)\big].

Even with exact entropy: this becomes $D_{\mathrm{KL}}(Q\Vert R)-D_{\mathrm{KL}}(P\Vert R)$ , not generally $D_{\mathrm{KL}}(Q\Vert P)$ .
If “reference number” means subtracting a fixed reference entropy: $h_R(Q)-c(P)$ simply shifts likelihood by a constant; it has exactly the same ranking as likelihood, and is not generally KL.
If it means matching both real-data statistics: the obvious alternative is the two-dimensional point $(h_R(Q),c(Q))$ relative to $(h_R(P),c(P))$ , or the following equal-weight summary:

A_C(Q,P;R)=|h_R(Q)-h_R(P)|+|c(Q)-c(P)|.

Interpretation: matches two scalar statistics; not a distributional distance. Both terms here use bits per GPT-2 source token; the unit weight is a baseline choice, not a fitted coefficient.
Results below: newly derived from existing full-bank measurements. These baselines were not the evidence that originally motivated the metric choice; they are added now to make the obvious alternatives explicit.

Likelihood & compression: baseline results

New arithmetic baseline from existing measurements: no new generation or GPU scoring; 113 configurations, including one real control; 1,024 sequences × 1,024 GPT-2 source tokens each.
Alignment: Qwen NLL and Zstandard collection rates come from the same saved full-bank summary; different budget from the 512-sequence historical comparison above.
Compressor: standalone collection coding, not the current per-document conditional W₁ metric.

Real-control quantity	Bits / GPT-2 source token
Qwen NLL	3.687758
Zstd collection rate	12.125656
NLL − Zstd rate	-8.437898

Sampler	NLL − rate above control	Median residual	Median residual margin	Median reference-matching sum
mirror	18/18	-0.93083	7.50707	7.73691
periodic	10/10	1.77367	10.21157	14.02695
phrase bank	27/27	-0.02848	8.40942	9.28190
topk iid	18/18	-1.71074	6.72716	6.72716

Diagnostic: the negative real-control residual demonstrates that these two practical coding systems do not yield a nonnegative KL estimate. Centering the residual at the real control does not repair the identity.
Reference-matching sum: zero for the real control by construction; a descriptive two-statistic matching score, not a calibrated detection test.

Inspect all 113 configurations · likelihood, rate and derived baselines

Setting	NLL	Zstd rate	NLL − rate	Residual margin	Reference-matching sum
{“kind”: “real”, “seed”: 0}	3.68776	12.12566	-8.43790	0.00000	0.00000
{“kind”: “topk_iid”, “seed”: 0, “size”: 32}	5.17602	6.27177	-1.09575	7.34215	7.34215
{“kind”: “topk_iid”, “seed”: 1, “size”: 32}	5.17338	6.26547	-1.09210	7.34580	7.34580
{“kind”: “topk_iid”, “seed”: 2, “size”: 32}	5.17419	6.26938	-1.09519	7.34271	7.34271
{“kind”: “topk_iid”, “seed”: 0, “size”: 64}	6.03124	7.35638	-1.32514	7.11276	7.11276
{“kind”: “topk_iid”, “seed”: 1, “size”: 64}	6.02754	7.35370	-1.32616	7.11174	7.11174
{“kind”: “topk_iid”, “seed”: 2, “size”: 64}	6.02883	7.35461	-1.32578	7.11212	7.11212
{“kind”: “topk_iid”, “seed”: 0, “size”: 128}	6.72232	8.31841	-1.59609	6.84180	6.84180
{“kind”: “topk_iid”, “seed”: 1, “size”: 128}	6.71529	8.31262	-1.59733	6.84056	6.84056
{“kind”: “topk_iid”, “seed”: 2, “size”: 128}	6.72040	8.31912	-1.59872	6.83918	6.83918
{“kind”: “topk_iid”, “seed”: 0, “size”: 256}	7.36431	9.19550	-1.83119	6.60671	6.60671
{“kind”: “topk_iid”, “seed”: 1, “size”: 256}	7.36611	9.18912	-1.82300	6.61489	6.61489
{“kind”: “topk_iid”, “seed”: 2, “size”: 256}	7.36470	9.18746	-1.82276	6.61514	6.61514
{“kind”: “topk_iid”, “seed”: 0, “size”: 512}	8.04064	10.09850	-2.05786	6.38004	6.38004
{“kind”: “topk_iid”, “seed”: 1, “size”: 512}	8.03564	10.09446	-2.05882	6.37908	6.37908
{“kind”: “topk_iid”, “seed”: 2, “size”: 512}	8.03906	10.08951	-2.05045	6.38745	6.38745
{“kind”: “topk_iid”, “seed”: 0, “size”: 1024}	8.75858	11.09309	-2.33452	6.10338	6.10338
{“kind”: “topk_iid”, “seed”: 1, “size”: 1024}	8.75599	11.08958	-2.33358	6.10432	6.10432
{“kind”: “topk_iid”, “seed”: 2, “size”: 1024}	8.75355	11.09061	-2.33706	6.10083	6.10083
{“kind”: “mirror”, “seed”: 0, “size”: 10}	2.22274	2.53220	-0.30946	8.12843	11.05847
{“kind”: “mirror”, “seed”: 1, “size”: 10}	2.21326	2.53114	-0.31787	8.12002	11.06902
{“kind”: “mirror”, “seed”: 2, “size”: 10}	2.21871	2.53288	-0.31417	8.12373	11.06182
{“kind”: “mirror”, “seed”: 0, “size”: 50}	3.09789	3.70358	-0.60569	7.83221	9.01194
{“kind”: “mirror”, “seed”: 1, “size”: 50}	3.10917	3.70134	-0.59217	7.84573	9.00290
{“kind”: “mirror”, “seed”: 2, “size”: 50}	3.10810	3.70415	-0.59605	7.84185	9.00116
{“kind”: “mirror”, “seed”: 0, “size”: 100}	3.46216	4.21992	-0.75777	7.68013	8.13133
{“kind”: “mirror”, “seed”: 1, “size”: 100}	3.45337	4.21964	-0.76626	7.67163	8.14041
{“kind”: “mirror”, “seed”: 2, “size”: 100}	3.45675	4.22076	-0.76401	7.67389	8.13591
{“kind”: “mirror”, “seed”: 0, “size”: 500}	4.21134	5.30826	-1.09691	7.34098	7.34098
{“kind”: “mirror”, “seed”: 1, “size”: 500}	4.20840	5.30379	-1.09540	7.34250	7.34250
{“kind”: “mirror”, “seed”: 2, “size”: 500}	4.20996	5.30914	-1.09918	7.33872	7.33872
{“kind”: “mirror”, “seed”: 0, “size”: 1000}	4.55782	5.82431	-1.26649	7.17141	7.17141
{“kind”: “mirror”, “seed”: 1, “size”: 1000}	4.55751	5.82372	-1.26621	7.17169	7.17169
{“kind”: “mirror”, “seed”: 2, “size”: 1000}	4.55738	5.82461	-1.26723	7.17067	7.17067
{“kind”: “mirror”, “seed”: 0, “size”: 5000}	5.34156	6.99243	-1.65087	6.78703	6.78703
{“kind”: “mirror”, “seed”: 1, “size”: 5000}	5.33654	6.99269	-1.65615	6.78175	6.78175
{“kind”: “mirror”, “seed”: 2, “size”: 5000}	5.34237	6.99640	-1.65403	6.78387	6.78387
{“kind”: “periodic”, “seed”: 0, “size”: 16}	0.20050	0.00245	0.19805	8.63595	15.61047
{“kind”: “periodic”, “seed”: 0, “size”: 32}	0.34652	0.00296	0.34356	8.78146	15.46393
{“kind”: “periodic”, “seed”: 0, “size”: 64}	0.63481	0.00391	0.63091	9.06880	15.17470
{“kind”: “periodic”, “seed”: 0, “size”: 100}	1.00359	0.00481	0.99878	9.43668	14.80502
{“kind”: “periodic”, “seed”: 0, “size”: 150}	1.52179	0.00588	1.51590	9.95380	14.28575
{“kind”: “periodic”, “seed”: 0, “size”: 200}	2.03834	0.00691	2.03143	10.46933	13.76816
{“kind”: “periodic”, “seed”: 0, “size”: 225}	2.31790	0.00745	2.31045	10.74835	13.48806
{“kind”: “periodic”, “seed”: 0, “size”: 256}	2.68302	0.00819	2.67483	11.11273	13.12221
{“kind”: “periodic”, “seed”: 0, “size”: 300}	3.16925	0.00912	3.16013	11.59803	12.63505
{“kind”: “periodic”, “seed”: 0, “size”: 400}	4.29281	0.01128	4.28154	12.71943	12.71943
{“kind”: “phrase_bank”, “seed”: 0, “size”: 10}	1.34224	1.44070	-0.09846	8.33944	13.03047
{“kind”: “phrase_bank”, “seed”: 1, “size”: 10}	1.33943	1.43905	-0.09962	8.33828	13.03493
{“kind”: “phrase_bank”, “seed”: 2, “size”: 10}	1.34158	1.43891	-0.09734	8.34056	13.03293
{“kind”: “phrase_bank”, “seed”: 0, “size”: 25}	1.98950	1.98059	0.00891	8.44681	11.84332
{“kind”: “phrase_bank”, “seed”: 1, “size”: 25}	1.98894	1.98008	0.00886	8.44676	11.84439
{“kind”: “phrase_bank”, “seed”: 2, “size”: 25}	1.99066	1.97702	0.01364	8.45154	11.84573
{“kind”: “phrase_bank”, “seed”: 0, “size”: 50}	2.41968	2.44816	-0.02848	8.40942	10.94557
{“kind”: “phrase_bank”, “seed”: 1, “size”: 50}	2.41433	2.44876	-0.03443	8.40347	10.95032
{“kind”: “phrase_bank”, “seed”: 2, “size”: 50}	2.42056	2.44828	-0.02772	8.41017	10.94458
{“kind”: “phrase_bank”, “seed”: 0, “size”: 100}	2.79336	2.89855	-0.10520	8.33270	10.12151
{“kind”: “phrase_bank”, “seed”: 1, “size”: 100}	2.79532	2.89943	-0.10410	8.33379	10.11866
{“kind”: “phrase_bank”, “seed”: 2, “size”: 100}	2.79302	2.90064	-0.10762	8.33027	10.11975
{“kind”: “phrase_bank”, “seed”: 0, “size”: 200}	3.22946	3.30206	-0.07259	8.36530	9.28190
{“kind”: “phrase_bank”, “seed”: 1, “size”: 200}	3.23413	3.29697	-0.06283	8.37506	9.28232
{“kind”: “phrase_bank”, “seed”: 2, “size”: 200}	3.23222	3.30043	-0.06822	8.36968	9.28077
{“kind”: “phrase_bank”, “seed”: 0, “size”: 500}	3.78257	3.85296	-0.07039	8.36751	8.36751
{“kind”: “phrase_bank”, “seed”: 1, “size”: 500}	3.78252	3.85431	-0.07179	8.36611	8.36611
{“kind”: “phrase_bank”, “seed”: 2, “size”: 500}	3.78606	3.85334	-0.06728	8.37061	8.37061
{“kind”: “phrase_bank”, “seed”: 0, “size”: 1000}	4.64636	4.26327	0.38309	8.82099	8.82099
{“kind”: “phrase_bank”, “seed”: 1, “size”: 1000}	4.65166	4.26942	0.38223	8.82013	8.82013
{“kind”: “phrase_bank”, “seed”: 2, “size”: 1000}	4.64960	4.26723	0.38237	8.82027	8.82027
{“kind”: “phrase_bank”, “seed”: 0, “size”: 2000}	5.02475	4.55526	0.46949	8.90739	8.90739
{“kind”: “phrase_bank”, “seed”: 1, “size”: 2000}	5.02671	4.55196	0.47474	8.91264	8.91264
{“kind”: “phrase_bank”, “seed”: 2, “size”: 2000}	5.01993	4.55055	0.46937	8.90727	8.90727
{“kind”: “phrase_bank”, “seed”: 0, “size”: 5000}	5.62397	4.96515	0.65882	9.09672	9.09672
{“kind”: “phrase_bank”, “seed”: 1, “size”: 5000}	5.62772	4.96535	0.66237	9.10027	9.10027
{“kind”: “phrase_bank”, “seed”: 2, “size”: 5000}	5.61571	4.97061	0.64510	9.08300	9.08300
{“kind”: “collapse”, “seed”: 0, “severity”: 0.01}	3.68573	12.01243	-8.32670	0.11120	0.11526
{“kind”: “collapse”, “seed”: 0, “severity”: 0.05}	3.68989	11.60495	-7.91506	0.52284	0.52284
{“kind”: “collapse”, “seed”: 0, “severity”: 0.1}	3.69184	11.07107	-7.37923	1.05867	1.05867
{“kind”: “collapse”, “seed”: 0, “severity”: 0.25}	3.68826	9.46410	-5.77584	2.66206	2.66206
{“kind”: “collapse”, “seed”: 0, “severity”: 0.5}	3.69081	6.66034	-2.96953	5.46837	5.46837
{“kind”: “collapse”, “seed”: 0, “severity”: 0.75}	3.71520	3.53115	0.18405	8.62195	8.62195
{“kind”: “collapse”, “seed”: 0, “severity”: 1.0}	3.72333	0.01979	3.70354	12.14144	12.14144
{“kind”: “shuffle”, “seed”: 0, “severity”: 0.01}	3.94579	12.22766	-8.28187	0.15603	0.36004
{“kind”: “shuffle”, “seed”: 0, “severity”: 0.05}	4.89779	12.62775	-7.72996	0.70794	1.71212
{“kind”: “shuffle”, “seed”: 0, “severity”: 0.1}	5.88418	13.05668	-7.17250	1.26539	3.12744
{“kind”: “shuffle”, “seed”: 0, “severity”: 0.25}	8.07508	13.97311	-5.89802	2.53987	6.23478
{“kind”: “shuffle”, “seed”: 0, “severity”: 0.5}	9.93534	14.76660	-4.83126	3.60663	8.88852
{“kind”: “shuffle”, “seed”: 0, “severity”: 0.75}	10.55573	15.00433	-4.44860	3.98930	9.74665
{“kind”: “shuffle”, “seed”: 0, “severity”: 1.0}	10.68650	15.03428	-4.34778	4.09012	9.90736
{“kind”: “pollute”, “seed”: 0, “severity”: 0.01}	3.88089	12.16368	-8.28279	0.15510	0.23115
{“kind”: “pollute”, “seed”: 0, “severity”: 0.05}	4.33490	12.19257	-7.85767	0.58023	0.71405
{“kind”: “pollute”, “seed”: 0, “severity”: 0.1}	4.71961	12.08189	-7.36228	1.07562	1.07562
{“kind”: “pollute”, “seed”: 0, “severity”: 0.25}	5.36461	11.32313	-5.95851	2.47938	2.47938
{“kind”: “pollute”, “seed”: 0, “severity”: 0.5}	5.14961	9.05305	-3.90344	4.53446	4.53446
{“kind”: “pollute”, “seed”: 0, “severity”: 0.75}	3.46072	5.57665	-2.11594	6.32196	6.77605
{“kind”: “pollute”, “seed”: 0, “severity”: 1.0}	15.14734	0.00318	15.14416	23.58206	23.58206
{“kind”: “repeat”, “seed”: 0, “severity”: 0.01}	3.69945	12.05163	-8.35217	0.08572	0.08572
{“kind”: “repeat”, “seed”: 0, “severity”: 0.05}	3.59342	11.65234	-8.05892	0.37897	0.56765
{“kind”: “repeat”, “seed”: 0, “severity”: 0.1}	3.42866	11.11212	-7.68346	0.75444	1.27263
{“kind”: “repeat”, “seed”: 0, “severity”: 0.25}	2.92069	9.44086	-6.52017	1.91773	3.45187
{“kind”: “repeat”, “seed”: 0, “severity”: 0.5}	2.06355	6.61859	-4.55504	3.88286	7.13127
{“kind”: “repeat”, “seed”: 0, “severity”: 0.75}	1.17444	3.76160	-2.58717	5.85073	10.87737
{“kind”: “repeat”, “seed”: 0, “severity”: 1.0}	0.16801	0.47700	-0.30899	8.12891	15.16841
{“kind”: “shuffle”, “region”: “prefix”, “seed”: 0, “severity”: 1.0}	5.15229	12.95435	-7.80207	0.63583	2.29323
{“kind”: “pollute”, “region”: “prefix”, “seed”: 0, “severity”: 1.0}	5.89448	9.93119	-4.03671	4.40119	4.40119
{“kind”: “shuffle”, “region”: “middle”, “seed”: 0, “severity”: 1.0}	5.24666	12.97459	-7.72793	0.70997	2.40783
{“kind”: “pollute”, “region”: “middle”, “seed”: 0, “severity”: 1.0}	3.12343	10.00423	-6.88080	1.55709	2.68575
{“kind”: “shuffle”, “region”: “suffix”, “seed”: 0, “severity”: 1.0}	5.17649	12.92838	-7.75189	0.68601	2.29145
{“kind”: “pollute”, “region”: “suffix”, “seed”: 0, “severity”: 1.0}	3.05608	9.96449	-6.90842	1.52948	2.79284
{“bank_size”: 1, “kind”: “bank”, “seed”: 0, “severity”: 1.0}	3.72333	0.01979	3.70354	12.14144	12.14144
{“bank_size”: 4, “kind”: “bank”, “seed”: 0, “severity”: 1.0}	4.20901	0.07863	4.13038	12.56828	12.56828
{“bank_size”: 16, “kind”: “bank”, “seed”: 0, “severity”: 1.0}	3.81066	0.26001	3.55065	11.98855	11.98855
{“bank_size”: 64, “kind”: “bank”, “seed”: 0, “severity”: 1.0}	3.63302	0.87098	2.76204	11.19994	11.30942
{“bank_size”: 256, “kind”: “bank”, “seed”: 0, “severity”: 1.0}	3.66501	3.19542	0.46959	8.90749	8.95299

Download unrounded baseline results and source hashes.

References

Download the bibliography (.bib)

[1]

C. E. Shannon, “A Mathematical Theory of Communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948, doi: 10.1002/j.1538-7305.1948.tb01338.x.

[2]

A. Franca and A. Tong, “Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics.” 2026. [Online]. Available: https://arxiv.org/abs/2606.08417

[3]

G. Kornhardt, M. Piening, J. Chemseddine, and G. Steidl, “SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics.” 2026. [Online]. Available: https://arxiv.org/abs/2610.06513

[4]

K. Pillutla et al., “MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers,” in Advances in Neural Information Processing Systems, 2021. [Online]. Available: https://arxiv.org/abs/2102.01454

[5]

Y. Collet and M. Kucherawy, “Zstandard Compression and the ‘application/zstd’ Media Type,” RFC Editor, techreport 8878, 2021. [Online]. Available: https://www.rfc-editor.org/rfc/rfc8878.html