Local LLM Field Notes — Evaluating Open-Weight Models on Apple Silicon
A first-hand friction log from running open-weight models locally on a MacBook Pro M4 Max with 128 GB of unified memory — GLM-4.5-Air in MLX-4bit and GGUF, the memory math that rules out the 750B-class models entirely, and the Ollama-versus-LM-Studio reality on Apple Silicon. Not many people have actually sat down and evaluated these specific models on this specific hardware. This is what actually happens when you do — the errors, the format-versus-engine traps, and the dimensions the “just run it locally” crowd forgets to price in.
🎯The one-line position
This isn't a “cloud good, local bad” take. It's the opposite of a lazy one: I ran the local path, hit the walls first-hand, and the walls are where the honest recommendation comes from. A recommendation to use the frontier API is only credible if you actually measured the alternative — so I did, on real hardware, and logged every place it bit.
⚖️The 10-second verdict
Seven dimensions, both stacks, no hedging. This is the summary the friction log below earns line by line.
📖Open source vs open weight vs closed — three different contracts
These three terms get used interchangeably, and they are not the same contract. What you can inspect, reproduce, and depend on differs at each tier — and most of what gets called “open source AI” is actually the middle column.
- You get: weights + training code + data recipe. The full pipeline is inspectable.
- You can: audit, reproduce, retrain from scratch, fork the whole lineage.
- Examples: OLMo 2 (AI2, Apache-2.0), Pythia (EleutherAI, Apache-2.0), BLOOM (BigScience, RAIL).
- You get: the trained weights to download and run. Training data and code stay private.
- You can: self-host, fine-tune, quantize — but never fully audit or reproduce how it was made.
- Examples: GLM-4.5-Air (MIT), DeepSeek-R1 (MIT), Qwen (mostly Apache-2.0), Llama (custom license), Mistral (Apache-2.0), Gemma (Google terms).
- You get: an API. Weights, code, and data all stay with the vendor.
- You can: build on capability + contractual terms — with zero infrastructure and no reproducibility at all.
- Examples: Claude, GPT-5 family, Gemini.
📓Friction log — dated, first-hand
The running log. Newest entries sit at the top; each one is a place the local path actually broke or surprised me, with the fix and the talking point it earned.
First conversation with GLM-4.5-Air: 16 minutes, 6,120 tokens, stopped by hand
With the runtimes fixed and the 56 GB model finally loaded, I asked one three-part question I already knew the answers to: the 750B memory math, a tokens/sec estimate for itself, and its own MoE active-parameter count. A self-grading exam. It was sent at 02:41:58. I pressed Stop at 02:57:53.
The scorecard: the 750B memory math was clean — 375 GB at 4-bit against 128 GB, “physically impossible,” correct. Everything else fell over, and the ways it fell over are the finding:
- 1.It doesn't know what it is. Mid-reasoning it decided it was “a 1.2B parameter model (as per the name ‘Air’, which typically indicates a smaller version)” — it is a 106B-parameter MoE with ~12B active. A model's training data predates the model itself; hosted APIs paper over this with an injected identity prompt. A raw checkpoint has no idea who it is, and every downstream answer inherited the error.
- 2.It contradicted itself — the summary said the dominant factor was “GPU compute power,” the body said “memory bandwidth.” (The body was right.)
- 3.It hallucinated arithmetic — a theoretical throughput of “60 × 10⁹ tokens/s.” Sixty billion tokens per second.
- 4.Then it ran away. After answering, it emitted a raw
<|user|>chat-template token, invented a follow-up question from a fictional user, renamed itself “GLM-4.5-Airlite,” answered its own invented question, repeated the summary twice, and was starting a third round when I stopped it. Stop-token and chat-template discipline is another ops surface you own locally — one the API world solved so long ago you forgot it existed.
<|user|> turns, the second summary, and at the very bottom the receipts: 7.21 tok/sec · 6,120 tokens · Stop reason: User Stopped — measured by the same window in which the model claimed “50–100 tokens/sec.”LM Studio: “No LM Runtime found for model format 'safetensors'!”
The 56 GB MLX-4bit model downloaded cleanly. Then LM Studio refused to load it: No LM Runtime found for model format ‘safetensors’. The model file sitting on disk meant nothing — there were zero inference engines installed. The first-launch runtime fetch had silently failed, so the app had a model and no way to run it. The fix is lms runtime from the CLI — pull the MLX and llama.cpp runtimes explicitly, and the same model loads instantly.
GLM-5.2 (750B class) does not fit in 128 GB
Before downloading anything, do the arithmetic: memory ≈ params × bytes-per-param. At a 4-bit quant, that's roughly 0.55 bytes per parameter. A 750B-parameter model is therefore on the order of 410 GB just for the weights — before context, activations, and KV cache. 128 GB of unified memory cannot hold it at any practical quant. “Open” does not mean “runnable.” The top of the open-weight leaderboard is a datacenter proposition — H100s, not a laptop.
The workable fallback — GLM-4.5-Air (106B MoE):
Ollama vs LM Studio — same model family, two stacks
GGUF via llama.cpp, headless, an OpenAI-compatible endpoint on :11434/v1, service-style UX. Great for scripting and headless / server use.
A GUI, MLX on Apple Silicon (noticeably faster decode on M-series thanks to Metal and unified memory), an OpenAI-compatible endpoint on :1234/v1.
New dated entries get appended to the top of this log as the evaluation continues.
🧭The dimensions people forget to ask about
When the ask is “evaluate open-source LLMs for us,” the conversation usually stops at benchmark scores and token cost. The dimensions below are where local-versus-hosted is actually decided — and they rarely make the first draft of anyone's comparison.
- ●Weights supply chain — tens of GB of numbers from a model hub; who attests they match the paper? Poisoned fine-tunes are a real class.
- ●Licensing — “open” spans a real range: GLM-4.5-Air is MIT — genuinely permissive, credit where due — while others ship Apache-2.0, custom acceptable-use (Llama-style), or research-only terms. The point isn't that open licensing is bad; it's that it's heterogeneous — legal review per model, per version.
- ●Alignment & safety — labs publish safety evals and stage rollouts; an open fine-tune has whatever its last fine-tuner gave it.
- ●Jurisdiction — some open frontier models carry procurement / geopolitical questions a US-based vendor doesn't.
Local weights are a depreciating asset. Hosted: the model improves under a stable API, and migration is a one-string change plus a re-run of your evals. Local: every release is a multi-GB re-download, a re-quantization, a fresh eval, and possibly an engine upgrade. Security patches to llama.cpp or MLX become your patch cadence.
The eval you ran last quarter is stale the day a better open model drops — and your switching cost is measured in downloads and ops, not a config line.
No deployed LLM self-learns. Weights are frozen at training time. A chat that “remembers” is doing context management — retrieval, memory files, caching — not learning. True for both local and hosted. What people actually mean, and the honest answer to each:
- “Adapt to our data” → RAG / context engineering. Works with either stack.
- “Get better over time” → hosted: the vendor ships better models. Local: you fine-tune (LoRA/QLoRA) — real, but an MLOps project with a data pipeline, eval harness, regression risk, and catastrophic-forgetting hazards.
- “Learn from each conversation” → a memory architecture (explicit, auditable stores) — an application-layer concern, not a model property.
A version of this argument made the rounds on a major tech podcast recently: to use a frontier API, you ship your data to the vendor with every session — and the vendor may learn from your usage and ship your workflow back to the world as a feature.
Concede what's true first. Data does cross the wire. And vendors do learn from aggregate usage patterns — which tools get called, where models fail, what people build. That's product telemetry, and it is how every platform (AWS, GitHub, Figma) has always evolved. The contractual line: enterprise API terms carry no-train commitments and bounded retention for your verbatim data. The distinction that matters is verbatim training (contractually barred) versus aggregate pattern-learning (real — and universal to SaaS).
The part the argument skips: the flywheel spins whether you join it or not. Opting out doesn't stop the vendor from learning — it removes you from the compounding, while everyone else's usage keeps improving models you no longer benefit from. Meanwhile the local weights you downloaded learn from no one — not even you. Six months later the frontier has compounded; your local model is exactly where you left it. You didn't protect your edge; you froze it.
The engineering answer is routing, not boycott: classify your data, keep the truly-sensitive workloads local (the legitimate data-gravity case), and ship the rest through contractual controls — the same committee/model-agnostic pattern as everywhere else in these notes.
Whichever stack you pick: no evals means no evaluation. Vibes are not a benchmark. Task-specific golden sets, regression runs on every model, quant, and engine change. And the silent trap: quantization is a quality knob — Q4 vs Q8 vs MLX-4bit of the same model are, for eval purposes, different models.
This is exactly the discipline the Observability & Evals survey is about; a local evaluation without a harness is just a demo.
- ●Local — hardware CapEx, electricity, and the expensive part: engineer time on ops. Zero marginal token cost.
- ●Hosted — linear per-token OpEx, zero idle cost, zero ops.
- ●Crossover exists only at sustained high volume with a stable model choice — which the depreciation problem actively undermines.
🟢Where local genuinely wins
Local isn't a loser — it's a specialist tool. It wins decisively in four situations, and pretending otherwise is as lazy as pretending it's always better.
✅Recommendation
The credible recommendation is the one that measured the alternative. Mine, in three moves.
Default to the frontier API for production
Frontier capability, a published safety posture, US jurisdiction, zero ops surface, continuous improvement under a stable API. Production velocity and trust are the whole game, and the API wins both.
Run the local eval anyway
Actually stand up GLM-4.5-Air on both Ollama and LM Studio/MLX and run it through the same eval harness. Not to deploy it — to earn the recommendation. A vendor recommendation with no measured alternative is an opinion; one with a local eval behind it is a finding.
Design provider-agnostic so the decision stays reversible
Wire the application to OpenAI-compatible endpoints and a model-routing layer so the local-versus-hosted choice is a config change, not a rewrite. Architecture is the hedge, not vendor loyalty. This is the Model-Agnostic Architecture posture — and it's why Mnemos, my local-first personal RAG, can swap between a local Ollama model and a frontier vendor per question with a single setting.