# aave-compress

**Requirements:** Python 3.10+, pip, no Docker/GPU/account needed (Anthropic API key optional, only for real Claude token counts) — full walkthrough in [SETUP.md](SETUP.md).

A deterministic, rule-based LLM prompt compressor that rewrites verbose Standard American English into AAVE (African American Vernacular English) grammatical patterns before the prompt is sent to a model. There is no decode step — the model reads the compressed form directly, the same way it already understands AAVE from its training data.

Every transform is grounded in documented AAVE grammar from the sociolinguistics literature — Labov, Rickford, Green, Fasold, and Baugh — not caricature. Full citations: [`docs/linguistics.md`](docs/linguistics.md). The pitch is not "AAVE is a shortcut" — several of these features (habitual *be*, completive *done*) make grammatical distinctions Standard English can't make without extra words. See the Linguistic Society of America's 1997 resolution, quoted in full in that doc, for why "slang" or "broken English" framing is factually wrong and this project avoids it.

## Honest status

**We do not publish a savings percentage without measuring it against realistic prompts using the target model's own tokenizer, and we haven't finished measuring yet.** A closely related project ("caveman") marketed 65–75% token savings that independent benchmarking found was really 14–21%. Full methodology and the actual (modest) numbers measured so far: [`docs/methodology.md`](docs/methodology.md). Short version: the only savings figure we currently stand behind is **5.0% mean at level `full` with `--strip-filler`**, measured on a realistic, non-cherry-picked 20-prompt corpus using GPT-4o's tiktoken — an approximation, NOT yet verified against Claude's own tokenizer. In that same run, individual prompts ranged from one that got 16.7% smaller down to several that didn't shrink at all. Without `--strip-filler` (AAVE markers only) it's more modest — roughly 2–3% mean depending on level (`full` = 3.1%, and one prompt actually got 3.3% *larger*). Every number here is a corpus-level mean or a stated range from the same run, never a single hand-picked prompt quoted as if it were the headline. Claude-specific numbers are not yet measured (requires `ANTHROPIC_API_KEY` in the build/eval environment). Grammatical compression and tokenizer-level compression are not automatically the same thing — that gap is exactly what `eval/run_eval.py` exists to catch honestly.

## Install

```bash
pip install -e ".[dev]"
```

Copy `env.example` to `.env` and fill in your own `ANTHROPIC_API_KEY` if you want real Claude token counts (the CLI and `eval/run_eval.py` load `.env` automatically via `python-dotenv`). Without it, the tool still works for compression itself — the key is only needed for Claude-accurate token counting, and `eval/run_eval.py --tiktoken-model gpt-4o` runs the (approximate) eval with no key at all.

```bash
cp env.example .env
```

## Usage

```bash
aave-compress "This endpoint usually times out when traffic is high." --level full

# Add --strip-filler to also remove hedges/politeness/filler before the AAVE
# markers. This is the configuration behind the 5.0% corpus figure above.
aave-compress "I was just wondering if you could maybe check the logs?" --level full --strip-filler
```

```python
from aave_compress.engine import compress

result = compress("This endpoint usually times out when traffic is high.", level="full")
print(result.compressed)  # "This endpoint be timin out when traffic high."
print(result.markers_applied)  # ["habitual_be", "zero_copula"]
```

Levels: `light` (zero copula only), `medium` (+ finna, steady), `full` (+ habitual be, completive done). See [`docs/linguistics.md`](docs/linguistics.md) for what each marker does and why, and [`docs/methodology.md`](docs/methodology.md) for how to run the eval harness yourself.

## Development

```bash
pytest              # 75 tests: rule-level hand-checked pairs + negative/guard cases, engine composition, tokenizer dispatch
python eval/run_eval.py --model claude-sonnet-5              # real Claude numbers (needs ANTHROPIC_API_KEY)
python eval/run_eval.py --tiktoken-model gpt-4o              # approximate GPT-family numbers (marker-only), no API key needed
python eval/run_eval.py --tiktoken-model gpt-4o --strip-filler  # reproduces the 5.0% mean (full+filler-strip) headline figure
```

`eval/results/` is gitignored — the eval writes its full per-prompt output there, but nothing in that directory ships in this repo. Regenerate it yourself with the commands above.

## Make it your own

- **Add a marker.** Grammar rules live in `src/aave_compress/rules.py`; each one is a self-contained transform with its own test cases in `tests/test_rules.py`. Follow the existing pattern (regex or token-level match → replacement → guard cases for when *not* to apply it) and cite the linguistics source in `docs/linguistics.md`.
- **Point the eval at your own corpus.** Swap `tests/fixtures/real_prompts.jsonl` for prompts that match what you actually send a model — the numbers in `docs/methodology.md` are only honest for the corpus they were measured on. Keep the same JSONL shape (`id`, `text`, `domain`).
- **Get real Claude numbers.** The GPT-4o figures above are an explicitly-labeled approximation. Set `ANTHROPIC_API_KEY` and run `python eval/run_eval.py --model claude-sonnet-5` to measure against the tokenizer that actually matters for your use case, then update `docs/methodology.md` with what you find — don't publish a savings claim you haven't measured.
- **Wire it into your own pipeline.** `compress()` in `src/aave_compress/engine.py` is a plain function — drop it in front of any place you're building a prompt string before it goes to a model.