aave-compress

Requirements: Python 3.10+, pip, no Docker/GPU/account needed (Anthropic API key optional, only for real Claude token counts) — full walkthrough in SETUP.md.

A deterministic, rule-based LLM prompt compressor that rewrites verbose Standard American English into AAVE (African American Vernacular English) grammatical patterns before the prompt is sent to a model. There is no decode step — the model reads the compressed form directly, the same way it already understands AAVE from its training data.

Every transform is grounded in documented AAVE grammar from the sociolinguistics literature — Labov, Rickford, Green, Fasold, and Baugh — not caricature. Full citations: docs/linguistics.md. The pitch is not "AAVE is a shortcut" — several of these features (habitual be, completive done) make grammatical distinctions Standard English can't make without extra words. See the Linguistic Society of America's 1997 resolution, quoted in full in that doc, for why "slang" or "broken English" framing is factually wrong and this project avoids it.

Honest status

We do not publish a savings percentage without measuring it against realistic prompts using the target model's own tokenizer, and we haven't finished measuring yet. A closely related project ("caveman") marketed 65–75% token savings that independent benchmarking found was really 14–21%. Full methodology and the actual (modest) numbers measured so far: docs/methodology.md. Short version: the only savings figure we currently stand behind is 5.0% mean at level full with --strip-filler, measured on a realistic, non-cherry-picked 20-prompt corpus using GPT-4o's tiktoken — an approximation, NOT yet verified against Claude's own tokenizer. In that same run, individual prompts ranged from one that got 16.7% smaller down to several that didn't shrink at all. Without --strip-filler (AAVE markers only) it's more modest — roughly 2–3% mean depending on level (full = 3.1%, and one prompt actually got 3.3% larger). Every number here is a corpus-level mean or a stated range from the same run, never a single hand-picked prompt quoted as if it were the headline. Claude-specific numbers are not yet measured (requires ANTHROPIC_API_KEY in the build/eval environment). Grammatical compression and tokenizer-level compression are not automatically the same thing — that gap is exactly what eval/run_eval.py exists to catch honestly.

Install

pip install -e ".[dev]"

Copy env.example to .env and fill in your own ANTHROPIC_API_KEY if you want real Claude token counts (the CLI and eval/run_eval.py load .env automatically via python-dotenv). Without it, the tool still works for compression itself — the key is only needed for Claude-accurate token counting, and eval/run_eval.py --tiktoken-model gpt-4o runs the (approximate) eval with no key at all.

cp env.example .env

Usage

aave-compress "This endpoint usually times out when traffic is high." --level full

# Add --strip-filler to also remove hedges/politeness/filler before the AAVE
# markers. This is the configuration behind the 5.0% corpus figure above.
aave-compress "I was just wondering if you could maybe check the logs?" --level full --strip-filler

from aave_compress.engine import compress

result = compress("This endpoint usually times out when traffic is high.", level="full")
print(result.compressed)  # "This endpoint be timin out when traffic high."
print(result.markers_applied)  # ["habitual_be", "zero_copula"]

Levels: light (zero copula only), medium (+ finna, steady), full (+ habitual be, completive done). See docs/linguistics.md for what each marker does and why, and docs/methodology.md for how to run the eval harness yourself.

Development

pytest              # 75 tests: rule-level hand-checked pairs + negative/guard cases, engine composition, tokenizer dispatch
python eval/run_eval.py --model claude-sonnet-5              # real Claude numbers (needs ANTHROPIC_API_KEY)
python eval/run_eval.py --tiktoken-model gpt-4o              # approximate GPT-family numbers (marker-only), no API key needed
python eval/run_eval.py --tiktoken-model gpt-4o --strip-filler  # reproduces the 5.0% mean (full+filler-strip) headline figure

eval/results/ is gitignored — the eval writes its full per-prompt output there, but nothing in that directory ships in this repo. Regenerate it yourself with the commands above.

Make it your own

• Add a marker. Grammar rules live in src/aave_compress/rules.py; each one is a self-contained transform with its own test cases in tests/test_rules.py. Follow the existing pattern (regex or token-level match → replacement → guard cases for when not to apply it) and cite the linguistics source in docs/linguistics.md.
• Point the eval at your own corpus. Swap tests/fixtures/real_prompts.jsonl for prompts that match what you actually send a model — the numbers in docs/methodology.md are only honest for the corpus they were measured on. Keep the same JSONL shape (id, text, domain).
• Get real Claude numbers. The GPT-4o figures above are an explicitly-labeled approximation. Set ANTHROPIC_API_KEY and run python eval/run_eval.py --model claude-sonnet-5 to measure against the tokenizer that actually matters for your use case, then update docs/methodology.md with what you find — don't publish a savings claim you haven't measured.
• Wire it into your own pipeline. compress() in src/aave_compress/engine.py is a plain function — drop it in front of any place you're building a prompt string before it goes to a model.