# Methodology — how we measure savings, and what we've actually measured so far

## Why this document exists

A closely related project, "caveman" (a Claude Code skill that rewrites output into terse, minimal-grammar English), marketed 65–75% token savings. Independent re-benchmarking — a public writeup by Kuba Guzik, and a separate real-traffic test against 614M tokens / $926 of actual Claude Code spend — found real savings of 14–21%, and as low as 3.7% once whole-session cache overhead was counted. The gap existed because the original claim was measured against artificially verbose baseline prompts rather than realistic ones.

We are not going to repeat that. This document states the rule up front: **no compression-percentage claim gets published anywhere in this project's README or marketing until it's produced by the process below, on a disclosed, non-cherry-picked prompt corpus, using the target model's own tokenizer.**

## The four requirements

1. **Model-correct token counting.** tiktoken (OpenAI's tokenizer) is not a valid stand-in for Claude token counts: a GPT-family tokenizer materially misestimates how many tokens a Claude model sees, and Claude's own tokenizer has changed across generations (newer Claude models, Opus 4.7 onward, produce meaningfully more tokens than earlier ones for identical text), so no fixed conversion factor is safe. The figures commonly cited for the tiktoken-vs-Claude gap are on the order of 15–20%+ on normal text and larger on code, with the cross-generation Claude change reported at roughly 30% — but **these specific percentages are approximate and still need a primary-source citation (a link to Anthropic's token-counting documentation plus the date it was checked) before they are quoted anywhere public.** That is the same disclosure bar this document sets for our own compression numbers; until it's met, treat them as directional, not exact. The technical decision does not depend on the exact figure: `count_tokens()` in this package refuses to fall back to tiktoken for any `claude-*` model — it raises rather than silently mismeasuring.
2. **A realistic, disclosed, non-cherry-picked prompt corpus.** `tests/fixtures/real_prompts.jsonl` — 20 original prompts across casual/planning/status/debugging/conversational registers, written to read the way people actually type to an assistant, not textbook-clean sentences. 5 of the 20 have zero applicable AAVE markers at any compression level, by design — an honest corpus includes prompts the tool doesn't help with.
3. **A ratio-vs-quality curve across compression levels** (`light`/`medium`/`full`), not one headline number.
4. **Report the whole distribution** (mean, min, max, and the zero-marker count), not just a favorable mean.

Run it yourself: `python eval/run_eval.py --model claude-sonnet-5` (requires `ANTHROPIC_API_KEY`) or `python eval/run_eval.py --tiktoken-model gpt-4o` for an approximate, explicitly-labeled GPT-family run with no API key. Full per-prompt output is written to `eval/results/latest.json` (gitignored — not for redistribution, regenerate it yourself).

## What we've actually measured so far

**Claude numbers: not yet measured.** No `ANTHROPIC_API_KEY` was available in the build environment. This is a known gap, not a hidden one — do not trust any Claude-specific savings number for this tool until someone runs `eval/run_eval.py --model claude-sonnet-5` with a real key and this section gets updated with the actual output.

**GPT-4o (tiktoken, approximate — NOT valid for Claude), run against the 20-prompt corpus:**

| level | zero-marker prompts | mean ratio | min ratio | max ratio | mean savings |
|---|---|---|---|---|---|
| light | 11/20 | 0.979 | 0.897 | 1.000 | 2.1% |
| medium | 6/20 | 0.971 | 0.897 | 1.000 | 2.9% |
| full | 5/20 | 0.969 | 0.893 | 1.033 | 3.1% |

(ratio = compressed_tokens / original_tokens; 1.000 = no change; 1.033 means the compressed version used *more* tokens than the original.)

Reproduce: `python eval/run_eval.py --tiktoken-model gpt-4o`

**GPT-4o (tiktoken, approximate — NOT valid for Claude), same 20-prompt corpus, WITH the filler-strip pass (`--strip-filler`) running before the AAVE markers:**

| level | zero-marker prompts | mean ratio | min ratio | max ratio | mean savings |
|---|---|---|---|---|---|
| light | 11/20 | 0.961 | 0.800 | 1.000 | 3.9% |
| medium | 6/20 | 0.952 | 0.800 | 1.000 | 4.8% |
| full | 5/20 | 0.950 | 0.833 | 1.000 | 5.0% |

Reproduce: `python eval/run_eval.py --tiktoken-model gpt-4o --strip-filler`

**This filler-strip `full` row (5.0% mean) is the only savings figure the project currently stands behind**, and it is still GPT-4o-tiktoken-approximate, NOT Claude-verified — the same caveats on the marker-only table apply here in full. In this run the best single prompt got 16.7% smaller (ratio 0.833) and, with filler removed first, no prompt got *larger* (max ratio 1.000). It is a corpus-level mean over 20 non-cherry-picked prompts, not a single favorable example — do not quote the 16.7% best-case as if it were the headline. Both tables above are copied from a manually-reviewed run and are regenerable from the commands shown; the raw per-prompt output (`eval/results/latest.json`) is gitignored, so regenerate it yourself rather than trusting a pasted number.

**Reading these numbers honestly (the bullets below read the marker-only table; the filler-strip table shifts the specifics — e.g. its best case is 16.7% and nothing got larger):**
- Real, realistic-corpus savings on this tokenizer are in the low single digits either way, not 50–70%. This matches the exact pattern that broke the caveman tool's credibility — a big claim collapses under a realistic, non-cherry-picked baseline.
- In the marker-only run, one prompt (`weekly-planning-post`) got measurably *worse*: "people **often** ask" → "people **be askin**" costs an extra GPT-4o token, because the specific subword the tokenizer produces for "askin" isn't more efficient than "ask" + "-ing" was. Grammatical compression and tokenizer-level compression are not the same thing, and this harness is built specifically to catch cases where they diverge.
- Best single-prompt result in the marker-only run: ~10.7% savings — real, but nowhere near a headline number, and it's the exception, not the mean.

## What this means for the project, honestly

The linguistic case for these five markers (documented in `linguistics.md`) is solid — they are genuinely doing more semantic work per morpheme than their SAE equivalents. What's *not* yet established is that this reliably translates into fewer tokens under a modern BPE tokenizer, where subword frequency in the training corpus — not word count — determines token count. "Askin" may simply not be a common enough subword to tokenize as cheaply as "ask" + "ing" does.

**Before publishing any savings claim in the README:**
1. Run the Claude-specific eval (`--model claude-sonnet-5`) once `ANTHROPIC_API_KEY` is available, and update the table above with real numbers.
2. Add a task-fidelity check — does Claude actually execute the compressed prompt the same way it executes the original? — before assuming a positive ratio is even a net win once correctness risk is priced in.
3. If the real number turns out to be single digits, say single digits. A modest, honestly-measured number that survives independent scrutiny is worth more than a headline that gets re-benchmarked down to a fraction of itself in public, the way caveman's did.