Local AI Setup

Requirements: a Mac with Apple Silicon (or Windows/Linux via the appendix), 8 GB+ RAM, Homebrew, no Docker/account/GPU/paid API needed — full walkthrough in SETUP.md.

Run real AI models — text, code, and voice — entirely on your own machine. No API key, no cloud account, no bill, no data leaving your hard drive.

This is a step-by-step guide for going from a stock Mac (Apple Silicon) to a working local AI stack you can chat with, call from scripts, and wire into agent tools like Claude Code. A Windows/Linux appendix is included at the end.

Why bother running AI locally

• Privacy. Nothing you type gets sent to a third party. This matters for anything sensitive: journaling, legal drafts, medical notes, business strategy, unreleased work.
• No metered cost. Once a model is downloaded, inference is free. Run it as many times as you want.
• No downtime, no rate limits, no policy changes. The model on your disk today works the same way in a year, regardless of what any provider does to their API or terms.
• Offline. Works on a plane, in a basement studio, anywhere without a connection.
• Ownership. You are not renting access to intelligence. You hold the weights.

The tradeoff: local models (especially on consumer hardware) are smaller and less capable than frontier cloud models like Claude. Treat this stack as your private, always-available layer — not a full replacement for frontier models on hard reasoning tasks.

What you need

• A Mac with Apple Silicon (M1/M2/M3/M4) is the easiest path — this guide is written for that first, with a Windows/Linux appendix at the end.
• At least 8 GB of unified memory (16 GB+ strongly recommended, 32 GB+ if you want to run larger models comfortably).
• Free disk space: budget 5-40 GB depending on how many models you keep around.

1. Install Ollama and run your first local model

Ollama is the simplest way to download and run open-weight language models locally. It handles the model file format, quantization, and gives you both a chat CLI and a local HTTP API.

Install

brew install ollama

No Homebrew? Download the .dmg installer directly from ollama.com and drag it to Applications instead.

Ollama runs as a background service. If it is not already running:

ollama serve

(If you installed via the .dmg, the menu-bar app starts this for you automatically.)

Pull a model

# A strong general-purpose small model
ollama pull llama3.2

# A model tuned for code
ollama pull qwen2.5-coder

Chat with it

ollama run llama3.2

This drops you into an interactive chat session in the terminal. Type /bye to exit.

Use it as an API

Ollama exposes an HTTP API on localhost:11434 the moment it's running — no extra setup.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Write a one-sentence description of a black hole.",
  "stream": false
}'

For chat-style calls:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "Explain quantization in one paragraph." }
  ],
  "stream": false
}'

This is the same API surface you will point other tools at later in this guide.

2. Choosing a model size for your RAM

Bigger models are smarter but slower and heavier. Match the model to your machine instead of guessing.

Pull a specific size with a tag:

ollama pull llama3.2:3b
ollama pull qwen2.5-coder:7b

Quantization, in plain language

A model's raw weights are normally stored as 16-bit numbers. Quantization rounds those numbers down to fewer bits (commonly 4-bit) to shrink the file and speed up inference, at a small cost in output quality.

• Ollama's default pulls are already quantized (usually to 4-bit, labeled Q4_K_M or similar) — you don't need to do anything extra to benefit from this.
• Rule of thumb: a 4-bit quantized model needs roughly (parameter count in billions) × 0.6 GB of RAM. An 8B model needs roughly 5 GB; a 14B model needs roughly 8-9 GB. Leave headroom for the OS and other apps.
• If a model feels sluggish or the machine starts swapping, drop down a size tier rather than fighting it — quality-per-second is usually better on the smaller model.

Check what's on disk any time:

ollama list

Remove a model you no longer need:

ollama rm llama3.2:3b

3. LM Studio — a GUI alternative to the terminal

LM Studio is a free desktop app for browsing, downloading, and chatting with local models with no command line required. Useful if you want a model-browser UI, side-by-side chat comparisons, or a visual way to try quantization levels before committing to a download.

1. Download LM Studio from lmstudio.ai and install it like any other Mac app.
2. Open it, use the built-in search to find a model (e.g. search "qwen2.5-coder"), and pick a quantization level from the list — the app shows the file size and an estimate of whether it fits your RAM before you download.
3. Click Download, then open the chat tab and select the model to start talking to it.
4. LM Studio can also run a local server (Developer tab → Start Server) that speaks the same OpenAI-compatible API format described in Section 5, on localhost:1234 by default.

Ollama and LM Studio can be installed side by side — they don't conflict, since Ollama defaults to port 11434 and LM Studio to port 1234.

4. Local voice: speech-to-text and text-to-speech

Speech-to-text with whisper.cpp

whisper.cpp is a fast, dependency-light C++ port of OpenAI's Whisper model that runs entirely offline, with Apple Silicon (Metal) acceleration built in.

brew install whisper-cpp

Download a model file (this is a one-time step; models range from ~75 MB to ~3 GB depending on accuracy needs):

mkdir -p ~/whisper-models
curl -L -o ~/whisper-models/ggml-base.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin

Transcribe an audio file:

whisper-cpp -m ~/whisper-models/ggml-base.en.bin -f your-audio.wav

whisper-cpp accepts WAV directly; convert other formats first with ffmpeg -i input.mp3 -ar 16000 output.wav.

Text-to-speech and voice cloning with Chatterbox via mlx-audio

mlx-audio runs audio models on Apple's MLX framework, using the Mac's Metal GPU. Chatterbox is an open-weight TTS model that supports voice cloning from a short reference clip.

python3 -m venv ~/mlx-audio-env
source ~/mlx-audio-env/bin/activate
pip install mlx-audio

Generate speech from text with a default voice:

python -m mlx_audio.tts.generate \
  --model mlx-community/chatterbox \
  --text "This is a locally generated voice." \
  --output out.wav

Clone a voice from a short reference sample (10-30 seconds of clean audio works best) by passing it as a reference:

python -m mlx_audio.tts.generate \
  --model mlx-community/chatterbox \
  --text "This is my cloned voice speaking." \
  --ref-audio path/to/reference-clip.wav \
  --output cloned-out.wav

Flags vary by mlx-audio version — run python -m mlx_audio.tts.generate --help to see the current options for your install.

Deactivate the virtual environment when done:

deactivate

5. Wiring local models into Claude Code and agent workflows

Ollama and LM Studio both expose an OpenAI-compatible chat completions endpoint. Any tool that lets you set a custom base URL and API key can be pointed at your local model instead of a cloud provider.

Ollama's OpenAI-compatible endpoint lives at:

http://localhost:11434/v1

LM Studio's local server (once started from the Developer tab) lives at:

http://localhost:1234/v1

Both accept any non-empty string as the API key field — there is no real authentication for a local server, but most client libraries require the field to be non-empty.

Example: calling it with the OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # any non-empty placeholder value works
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Summarize this guide in one sentence."}],
)

print(response.choices[0].message.content)

Wiring it into agent frameworks

Most agent frameworks and coding-agent tools that support "custom OpenAI-compatible endpoint" or "custom model provider" settings will work here — point the base URL at your local Ollama or LM Studio server and set the model name to whatever you pulled (e.g. llama3.2 or qwen2.5-coder).

This is useful for:

• Running a cheap/free local model as a first-pass filter before escalating to a frontier model for hard cases.
• Offline development and testing of agent logic without burning API credits.
• Any workflow step that doesn't need frontier-level reasoning (simple extraction, formatting, boilerplate generation).

See config.example.env in this repo for the environment variable pattern to drop into a project's .env file.

6. Make it yours

• Swap in whatever models fit your hardware and use case — the commands above work identically for any model in the Ollama library (ollama.com/library) or on Hugging Face for LM Studio/mlx-audio.
• Automate model pulls with scripts/pull-models.sh in this repo — edit the model list at the top of the file first.
• If you build a script or agent that talks to your local model, keep the base URL and model name in an environment variable (see config.example.env) rather than hardcoding it, so you can swap providers later without touching code.
• Chain steps together: transcribe audio with whisper.cpp, send the transcript to a local Ollama model for summarization, then speak the summary back out with Chatterbox — all offline, all free after setup.

Appendix: Windows and Linux

See docs/windows-linux.md for install instructions on Windows and Linux. The Ollama chat/API workflow (Sections 1-2) and the OpenAI-compatible wiring (Section 5) work identically on all three platforms once Ollama is installed — only the install command and the voice-stack setup (Section 4) differ.

Repo contents

README.md                  - this guide
SETUP.md                   - requirements checklist + step-by-step install
config.example.env         - environment variable template for Section 5
scripts/install-mac.sh     - installs Ollama on macOS
scripts/pull-models.sh     - pulls a starter set of models, edit the list at the top
docs/windows-linux.md      - Windows/Linux install appendix