# Local AI Setup

**Requirements:** a Mac with Apple Silicon (or Windows/Linux via the appendix), 8 GB+ RAM, Homebrew, no Docker/account/GPU/paid API needed — full walkthrough in [SETUP.md](SETUP.md).

Run real AI models — text, code, and voice — entirely on your own machine. No API key, no cloud account, no bill, no data leaving your hard drive.

This is a step-by-step guide for going from a stock Mac (Apple Silicon) to a working local AI stack you can chat with, call from scripts, and wire into agent tools like Claude Code. A Windows/Linux appendix is included at the end.

## Why bother running AI locally

- **Privacy.** Nothing you type gets sent to a third party. This matters for anything sensitive: journaling, legal drafts, medical notes, business strategy, unreleased work.
- **No metered cost.** Once a model is downloaded, inference is free. Run it as many times as you want.
- **No downtime, no rate limits, no policy changes.** The model on your disk today works the same way in a year, regardless of what any provider does to their API or terms.
- **Offline.** Works on a plane, in a basement studio, anywhere without a connection.
- **Ownership.** You are not renting access to intelligence. You hold the weights.

The tradeoff: local models (especially on consumer hardware) are smaller and less capable than frontier cloud models like Claude. Treat this stack as your private, always-available layer — not a full replacement for frontier models on hard reasoning tasks.

## What you need

- A Mac with Apple Silicon (M1/M2/M3/M4) is the easiest path — this guide is written for that first, with a Windows/Linux appendix at the end.
- At least 8 GB of unified memory (16 GB+ strongly recommended, 32 GB+ if you want to run larger models comfortably).
- Free disk space: budget 5-40 GB depending on how many models you keep around.

---

## 1. Install Ollama and run your first local model

[Ollama](https://ollama.com) is the simplest way to download and run open-weight language models locally. It handles the model file format, quantization, and gives you both a chat CLI and a local HTTP API.

### Install

```bash
brew install ollama
```

No Homebrew? Download the `.dmg` installer directly from ollama.com and drag it to Applications instead.

Ollama runs as a background service. If it is not already running:

```bash
ollama serve
```

(If you installed via the `.dmg`, the menu-bar app starts this for you automatically.)

### Pull a model

```bash
# A strong general-purpose small model
ollama pull llama3.2

# A model tuned for code
ollama pull qwen2.5-coder
```

### Chat with it

```bash
ollama run llama3.2
```

This drops you into an interactive chat session in the terminal. Type `/bye` to exit.

### Use it as an API

Ollama exposes an HTTP API on `localhost:11434` the moment it's running — no extra setup.

```bash
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Write a one-sentence description of a black hole.",
  "stream": false
}'
```

For chat-style calls:

```bash
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "Explain quantization in one paragraph." }
  ],
  "stream": false
}'
```

This is the same API surface you will point other tools at later in this guide.

---

## 2. Choosing a model size for your RAM

Bigger models are smarter but slower and heavier. Match the model to your machine instead of guessing.

| Unified RAM | Model size to target | Example |
|---|---|---|
| 8 GB | 3B parameters | `llama3.2:3b` |
| 16 GB | 7B-8B parameters | `llama3.1:8b`, `qwen2.5-coder:7b` |
| 32 GB+ | 14B+ parameters | `qwen2.5:14b`, `qwen2.5-coder:32b` |

Pull a specific size with a tag:

```bash
ollama pull llama3.2:3b
ollama pull qwen2.5-coder:7b
```

### Quantization, in plain language

A model's raw weights are normally stored as 16-bit numbers. Quantization rounds those numbers down to fewer bits (commonly 4-bit) to shrink the file and speed up inference, at a small cost in output quality.

- Ollama's default pulls are already quantized (usually to 4-bit, labeled `Q4_K_M` or similar) — you don't need to do anything extra to benefit from this.
- Rule of thumb: a 4-bit quantized model needs roughly (parameter count in billions) × 0.6 GB of RAM. An 8B model needs roughly 5 GB; a 14B model needs roughly 8-9 GB. Leave headroom for the OS and other apps.
- If a model feels sluggish or the machine starts swapping, drop down a size tier rather than fighting it — quality-per-second is usually better on the smaller model.

Check what's on disk any time:

```bash
ollama list
```

Remove a model you no longer need:

```bash
ollama rm llama3.2:3b
```

---

## 3. LM Studio — a GUI alternative to the terminal

[LM Studio](https://lmstudio.ai) is a free desktop app for browsing, downloading, and chatting with local models with no command line required. Useful if you want a model-browser UI, side-by-side chat comparisons, or a visual way to try quantization levels before committing to a download.

1. Download LM Studio from lmstudio.ai and install it like any other Mac app.
2. Open it, use the built-in search to find a model (e.g. search "qwen2.5-coder"), and pick a quantization level from the list — the app shows the file size and an estimate of whether it fits your RAM before you download.
3. Click Download, then open the chat tab and select the model to start talking to it.
4. LM Studio can also run a local server (Developer tab → Start Server) that speaks the same OpenAI-compatible API format described in Section 5, on `localhost:1234` by default.

Ollama and LM Studio can be installed side by side — they don't conflict, since Ollama defaults to port 11434 and LM Studio to port 1234.

---

## 4. Local voice: speech-to-text and text-to-speech

### Speech-to-text with whisper.cpp

[whisper.cpp](https://github.com/ggml-org/whisper.cpp) is a fast, dependency-light C++ port of OpenAI's Whisper model that runs entirely offline, with Apple Silicon (Metal) acceleration built in.

```bash
brew install whisper-cpp
```

Download a model file (this is a one-time step; models range from ~75 MB to ~3 GB depending on accuracy needs):

```bash
mkdir -p ~/whisper-models
curl -L -o ~/whisper-models/ggml-base.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin
```

Transcribe an audio file:

```bash
whisper-cpp -m ~/whisper-models/ggml-base.en.bin -f your-audio.wav
```

`whisper-cpp` accepts WAV directly; convert other formats first with `ffmpeg -i input.mp3 -ar 16000 output.wav`.

### Text-to-speech and voice cloning with Chatterbox via mlx-audio

[mlx-audio](https://github.com/Blaizzy/mlx-audio) runs audio models on Apple's MLX framework, using the Mac's Metal GPU. [Chatterbox](https://github.com/resemble-ai/chatterbox) is an open-weight TTS model that supports voice cloning from a short reference clip.

```bash
python3 -m venv ~/mlx-audio-env
source ~/mlx-audio-env/bin/activate
pip install mlx-audio
```

Generate speech from text with a default voice:

```bash
python -m mlx_audio.tts.generate \
  --model mlx-community/chatterbox \
  --text "This is a locally generated voice." \
  --output out.wav
```

Clone a voice from a short reference sample (10-30 seconds of clean audio works best) by passing it as a reference:

```bash
python -m mlx_audio.tts.generate \
  --model mlx-community/chatterbox \
  --text "This is my cloned voice speaking." \
  --ref-audio path/to/reference-clip.wav \
  --output cloned-out.wav
```

Flags vary by mlx-audio version — run `python -m mlx_audio.tts.generate --help` to see the current options for your install.

Deactivate the virtual environment when done:

```bash
deactivate
```

---

## 5. Wiring local models into Claude Code and agent workflows

Ollama and LM Studio both expose an **OpenAI-compatible** chat completions endpoint. Any tool that lets you set a custom base URL and API key can be pointed at your local model instead of a cloud provider.

Ollama's OpenAI-compatible endpoint lives at:

```
http://localhost:11434/v1
```

LM Studio's local server (once started from the Developer tab) lives at:

```
http://localhost:1234/v1
```

Both accept any non-empty string as the API key field — there is no real authentication for a local server, but most client libraries require the field to be non-empty.

### Example: calling it with the OpenAI Python SDK

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # any non-empty placeholder value works
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Summarize this guide in one sentence."}],
)

print(response.choices[0].message.content)
```

### Wiring it into agent frameworks

Most agent frameworks and coding-agent tools that support "custom OpenAI-compatible endpoint" or "custom model provider" settings will work here — point the base URL at your local Ollama or LM Studio server and set the model name to whatever you pulled (e.g. `llama3.2` or `qwen2.5-coder`).

This is useful for:

- Running a cheap/free local model as a first-pass filter before escalating to a frontier model for hard cases.
- Offline development and testing of agent logic without burning API credits.
- Any workflow step that doesn't need frontier-level reasoning (simple extraction, formatting, boilerplate generation).

See `config.example.env` in this repo for the environment variable pattern to drop into a project's `.env` file.

---

## 6. Make it yours

- Swap in whatever models fit your hardware and use case — the commands above work identically for any model in the Ollama library (`ollama.com/library`) or on Hugging Face for LM Studio/mlx-audio.
- Automate model pulls with `scripts/pull-models.sh` in this repo — edit the model list at the top of the file first.
- If you build a script or agent that talks to your local model, keep the base URL and model name in an environment variable (see `config.example.env`) rather than hardcoding it, so you can swap providers later without touching code.
- Chain steps together: transcribe audio with whisper.cpp, send the transcript to a local Ollama model for summarization, then speak the summary back out with Chatterbox — all offline, all free after setup.

## Appendix: Windows and Linux

See `docs/windows-linux.md` for install instructions on Windows and Linux. The Ollama chat/API workflow (Sections 1-2) and the OpenAI-compatible wiring (Section 5) work identically on all three platforms once Ollama is installed — only the install command and the voice-stack setup (Section 4) differ.

## Repo contents

```
README.md                  - this guide
SETUP.md                   - requirements checklist + step-by-step install
config.example.env         - environment variable template for Section 5
scripts/install-mac.sh     - installs Ollama on macOS
scripts/pull-models.sh     - pulls a starter set of models, edit the list at the top
docs/windows-linux.md      - Windows/Linux install appendix
```