AIChatKitLlama adds on-device GGUF inference via llama.cpp to any app already using AIChatKit. Models run entirely in-process using Metal GPU acceleration — no network calls after the initial model download.
Tested with Gemma 4, Llama 3.x, Mistral/Mixtral, and Phi-3/Phi-4 — any GGUF model compatible with llama.cpp works. Chat templates for common families are applied automatically.
nGpuLayers: 99 puts every layer on Metal GPU; -1 forces CPU-only. Tune contextSize (KV cache) and maxTurns (older-turn truncation) per model and device.
LlamaProvider is a Swift actor — the model loads from disk on the first stream() or complete() call and stays resident in memory, with calls naturally serialized.
temperature, topP, topK, minP, and penaltyRepeat are all exposed via ChatRequestOptions. Cloud providers silently ignore the llama-specific parameters, so the same options struct works across every provider.
LlamaProvider implements the same AIChatCore protocol as the cloud and MLX providers — no separate integration path for local-model chat.
.package(url: "https://github.com/NerdSnipe-Inc/AIChatKit", from: "0.1.0"),
.package(url: "https://github.com/NerdSnipe-Inc/AIChatKitLlama", from: "0.1.0").product(name: "AIChatCore", package: "AIChatKit"),
.product(name: "AIChatUI", package: "AIChatKit"),
.product(name: "AIChatLlama", package: "AIChatKitLlama")AIChatLlama pulls a ~500 MB llama.cpp XCFramework (via llama.swift). Add it only to targets that need local inference, and add AIChatKitLlama's resolved package to .gitignore rather than committing it.
llama.cpp's GGUF format has the broadest model compatibility of any on-device runtime, at the cost of a large (~500 MB) bundled binary. Apps that only need Apple-native models should look at AIChatKitMLX instead — it has no binary framework to ship.
AIChatKitMLX runs Apple MLX models and requires no bundled binary, but is limited to the mlx-community model ecosystem. AIChatKitLlama runs any GGUF model via llama.cpp at the cost of a ~500 MB XCFramework. Use MLX for Apple-native models, Llama for broader model compatibility.
Hugging Face (huggingface.co/models?library=gguf). Q4_K_M quantization is a good balance of quality and size for most use cases.
Only for the initial model download. After that, LlamaProvider runs entirely in-process with no network calls.
Add AIChatKitLlama alongside AIChatKit and point LlamaProvider at any GGUF model — same ChatSession API you already use for cloud providers.