1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106 | # Setup
`aave-compress` is a pure-Python package (Python 3.10+). No system binaries, no browser drivers, no compiled dependencies. It works fully offline for the compression feature itself; an Anthropic API key is only needed if you want real Claude token counts.
## What you need
| Requirement | Why | Have it already? |
|---|---|---|
| Python 3.10 or newer | `pyproject.toml` sets `requires-python = ">=3.10"`; the code uses modern type-hint syntax (`str \| None`) that needs 3.10+ | Check: `python3 --version` |
| pip (comes with Python) | Installs the package and its dependencies | Check: `python3 -m pip --version` |
| `anthropic` Python package | Required dependency (`pyproject.toml`); used for real Claude token counting via `client.messages.count_tokens` in `src/aave_compress/tokenizers.py` | Installed automatically by the install command below |
| `ANTHROPIC_API_KEY` (optional) | Only needed if you want real Claude token counts. Compression itself (`compress()`, the CLI) works with no key at all. | Get one at console.anthropic.com if you want this |
| `tiktoken` (optional, dev extra) | Approximate GPT-family token counting (`--tiktoken-model gpt-4o`), explicitly labeled as NOT valid for Claude models | Installed with `pip install -e ".[dev]"` |
| `pytest` (optional, dev extra) | Runs the 75-test suite | Installed with `pip install -e ".[dev]"` |
| `python-dotenv` (optional, dev extra) | Auto-loads a `.env` file so you don't have to `export` the API key manually | Installed with `pip install -e ".[dev]"` |
**NOT needed:** Docker (this is a plain Python package, no containers involved). No GPU. No database. No account/signup to use the tool itself โ an Anthropic account is only needed if you want real Claude token counts, and the tool works without it. Not macOS-only โ pure Python, runs on macOS, Linux, and Windows identically.
## Install, step by step
1. **Install Python 3.10+ if you don't have it.**
macOS (Homebrew):
```bash
brew install python@3.12
```
If you don't have Homebrew yet, install it first:
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```
Windows: download the installer from python.org/downloads and check "Add python.exe to PATH" during install.
Linux (Debian/Ubuntu):
```bash
sudo apt update && sudo apt install -y python3 python3-pip python3-venv
```
Check it worked:
```bash
python3 --version
```
Expected output: `Python 3.10.x` or higher (e.g. `Python 3.12.4`).
2. **Clone the repo.**
```bash
git clone <this-repo-url>
cd aave-compress
```
Check it worked: `ls` should show `pyproject.toml`, `src/`, `tests/`, `eval/`.
3. **(Recommended) Create a virtual environment**, so this package's dependencies don't mix with other Python projects on your machine.
```bash
python3 -m venv .venv
source .venv/bin/activate
```
Windows (cmd/PowerShell): `.venv\Scripts\activate`
Check it worked: your shell prompt should now be prefixed with `(.venv)`.
4. **Install the package in editable mode, with the dev extras** (pytest, tiktoken, python-dotenv).
```bash
pip install -e ".[dev]"
```
What it does: installs `aave-compress` itself plus its required dependency (`anthropic`) and the optional dev tools, and registers the `aave-compress` command-line tool.
Check it worked:
```bash
aave-compress --help
```
Expected output: usage text starting with `usage: aave-compress [-h] [--level {full,light,medium}] ...`
## Configure
1. Copy the example env file:
```bash
cp env.example .env
```
2. Open `.env` and, only if you want real Claude token counts, replace the placeholder with your own key:
```
ANTHROPIC_API_KEY=sk-ant-YOUR_KEY_HERE # keyzhub-allow
```
Get a key at console.anthropic.com if you don't have one. Never commit `.env` โ it's already in `.gitignore`.
If you skip this step entirely, the tool still works: compression runs with no key, and `python eval/run_eval.py --tiktoken-model gpt-4o` gives approximate GPT-family numbers with no key required. You only lose real Claude-accurate token counts.
## Run it
The simplest first command:
```bash
aave-compress "This endpoint usually times out when traffic is high." --level full
```
What success looks like: it prints `Level: full`, the original text, a compressed version (something like `This endpoint be timin out when traffic high.`), and a list of markers applied (e.g. `habitual_be, zero_copula`). If you did not set `ANTHROPIC_API_KEY`, it will also print a note to stderr saying token counts are unavailable โ that's expected, not an error.
Run the test suite to confirm the install is fully healthy:
```bash
pytest
```
Expected output: a line ending in `75 passed` (approximately โ check the actual count) with no failures.
## Troubleshooting
1. **`aave-compress: command not found`** โ the install didn't register the CLI entry point, usually because you're not in the virtual environment you installed into, or you ran `pip install` instead of `pip install -e ".[dev]"`. Fix: `source .venv/bin/activate` (re-activate), then re-run `pip install -e ".[dev]"`.
2. **`ModuleNotFoundError: No module named 'anthropic'` (or `tiktoken`, `dotenv`)** โ you installed without the `[dev]` extra, or you're running a different Python/pip than the one you installed into (common when a system Python and a venv Python both exist). Fix: confirm `which python3` and `which pip` point inside `.venv/`, then run `pip install -e ".[dev]"` again.
3. **`RuntimeError: ANTHROPIC_API_KEY is not set...` when running the CLI or `eval/run_eval.py`** โ this is not a crash, it's the tool refusing to invent a fake Claude token count (see `src/aave_compress/tokenizers.py`). Fix: either set `ANTHROPIC_API_KEY` in `.env` (see Configure above) and make sure you're running from a directory where `python-dotenv` can find that `.env`, or pass `--tiktoken-model gpt-4o` for an approximate GPT-family count that needs no key.
|