Today I Learned: Genuinely Capable AI Now Runs on My Laptop

Let me be honest up front: a year ago I assumed "run the model on your own machine" meant accepting something noticeably dumber than the cloud. Useful for a demo, not for real work. So I mostly didn’t bother.

This week I sat down and actually tried it, and I had to update that belief. On June 5, 2026, Google DeepMind shipped Quantization-Aware Training (QAT) checkpoints for the Gemma 4 family. Here’s the short version of what I learned, and the one distinction it took me a while to get straight: dropping the weights to 4-bit is what shrinks the memory footprint to a fraction of full precision, enough that a 26B model fits on a 16GB laptop. The new trick, QAT, is what keeps the quality close to the full-precision model while it’s that small. (Independent testers like Unsloth measured roughly 72% less memory at 4-bit; Google doesn’t put a single percentage on it, but its own size table, which I’ll show, backs up the gist.) I’m writing this up the way I’d explain it to a teammate who hasn’t had time to dig in yet. Where a number is contested or I’m taking it second-hand, I flag it rather than smooth it over.

Here’s the bit that made it real for me: I ran it in LM Studio on my own hardware, a MacBook Pro, Apple M5 Pro, 24 GB. Not a benchmark rig. The laptop I work on every day.

Two of the three offloaded fully to the GPU. The 31B only goes partial on my 24 GB of unified memory, so it’s slower. But even partial, it opens a surprising number of doors in my workflow. That’s the moment local stopped being a toy for me.

Teaching a model to survive being shrunk

Here’s the mental model that finally made it click for me. To run a big model on a small machine, you quantize it: store the weights at lower precision (4-bit instead of 16-bit), so it takes far less memory. The naive way is to train normally and then round everything down afterwards. That’s Post-Training Quantization (PTQ), and it can quietly cost you quality, because the model never “knew” it would be compressed.

QAT flips the order. It simulates the 4-bit math during training, so the weights learn to tolerate the compression while they’re still being shaped. Google’s framing: “by simulating quantization during training, QAT minimizes quality loss when the model is compressed.” They even claim it can land higher overall quality than a standard PTQ baseline. I can’t independently confirm that number. But the QAT builds I ran genuinely didn’t feel like the lobotomized local models I remember.

Wait, a 26B model on a 16GB laptop?

This is the number that surprised me most. Here’s the real on-disk-into-memory footprint across the family. I’m plotting the practical footprint: the 4-bit weights plus the ~20% loading overhead Google notes (and KV cache, in practice). Not the headline weight size, because that overhead is exactly what decided whether a model fit on my Mac.

The thing nobody told me until I hit it myself: on Apple Silicon, the GPU can’t use every byte of your unified memory. macOS keeps a slice for the system. In practice that left me with roughly ~18 GB usable on my 24 GB machine (call it a ceiling, not an exact figure). That’s what explained most of what I saw:

• 12B (~8 GB): comfortably under both lines. Full offload, fast, my daily driver.
• 26B-A4B (~17 GB): slips under the ~18 GB ceiling, so it fully offloads on my Mac. It’s a Mixture-of-Experts model that only activates 3.8B parameters per token, which is why it punches so far above its memory weight.
• 31B (~21 GB): crosses the ceiling, so some layers stay on the CPU. Partial offload, noticeably slower, but still genuinely useful.

One caveat the chart can’t show, because it’s scoped to my machine: a smaller 16 GB laptop has even less usable GPU than I do, so the same 26B-A4B is right at the edge there. The official numbers say it loads in ~14–15 GB, which fits, but only with short context and not much else running. “26B on a 16GB laptop” is real; it’s just tight. The extra room on 24 GB is what made it feel relaxed rather than precarious.

But is it actually any good?

Fitting on the laptop means nothing if the output is weak. So I went looking at the published numbers, and the generational jump from Gemma 3 to Gemma 4 is the part that genuinely changed my mind:

Look at the reasoning and coding columns. GPQA Diamond goes from 42.4% → 84.3%. LiveCodeBench from 29.1% → 80.0%. AIME from 20.8% → 89.2%. Big jumps. But I want to be straight about one thing, because it tripped me up: Google’s table runs Gemma 4 in thinking mode and the Gemma 3 baseline without it, so part of that gap (AIME especially, where just letting a model reason step-by-step is worth a lot) is the mode, not purely the new model. Even discounting for that, it’s a real generational step, and it runs on my desk.

The real unlock isn’t speed. It’s where the data lives

I work in security, so the moment something is “good enough locally,” my brain immediately goes to all the things I currently can’t send to a cloud API: vulnerability findings, internal security docs, half-finished security questionnaires, anything with customer data in it. With local inference, that data simply never leaves hardware I control.

The way I now understand it: every cloud AI call carrying personal data is potentially a cross-border transfer, which under GDPR pulls in Article 28 (you need a Data Processing Agreement) and Articles 44–46 (transfer safeguards). And because of Schrems II and the US CLOUD Act, even “EU region” hosting from a US-headquartered vendor carries residual jurisdictional risk. Keeping the data on my Mac is, quite literally, the architectural version of Article 25: “data protection by design and by default.”

The cautionary tale that made this concrete for me is Samsung, 2023: engineers pasted semiconductor source code and an internal meeting transcript into ChatGPT. Three leaks in 20 days. The company ended up limiting uploads and threatening to block the tool entirely. That’s OWASP’s LLM02: Sensitive Information Disclosure, which, per OWASP’s 2025 report, made the biggest jump on the list, from #6 to #2. Local inference is the cleanest way I know to make that whole class of accident impossible: there’s no third party to leak to.

The part I had to slow down on: local isn’t automatically safe

This is where I had to check my own excitement. “Local” solves the data-residency problem. It does not hand you a secure system. A few things I learned the hard way (by reading, thankfully, not by getting burned):

• Prompt injection still applies, identically. It’s still OWASP’s #1 LLM risk (LLM01). Running locally changes nothing here. If your model reads untrusted text and can take actions, you have the same problem.
• Model weights are a supply chain. Researchers have shipped surgically poisoned models (PoisonGPT) that pass standard benchmarks, and in July 2025 Pillar Security disclosed “Poisoned GGUF Templates”: malicious logic hidden in the chat template that the repository UI doesn’t show you, across a format Pillar reported spans over 1.5 million files on public hubs.
• You own the host. An Ollama server bound to 0.0.0.0 with no auth is reachable across your whole network. And you’re now the one patching the runtime. Nobody pushes a fix for you.

Going local changes the threat model. It doesn’t delete it.

The plan I’m giving myself

Not advice from on high, just the staged approach I’ve settled on after this week:

1. Pilot on the boring-but-sensitive stuff. Run the 12B QAT for the work I currently refuse to send to a cloud API: drafting questionnaire answers, summarizing internal docs, triaging and de-duping vuln findings, private RAG over internal documentation. Gemma 4 is Apache 2.0, so commercial use is fine.
2. Harden the host. Treat the model as untrusted: download weights only from official repos, verify the hash, pin versions, never expose the API on 0.0.0.0, and keep the runtime patched. For the most sensitive work, go air-gapped after the download.
3. Govern it. Going local doesn’t exempt me from a DPIA, a lawful basis, retention limits, or breach procedures. The EU AI Act’s high-risk obligations land August 2, 2026, so it’s worth mapping use against those categories now and leaning on NIST AI 600-1 and the OWASP LLM Top 10 for actual controls.

The honest loose ends

I don’t want to pretend this is settled. A few things I’m genuinely still chewing on:

• Frontier-grade tasks. For the hardest coding or agentic work, a local 12B may still trail hosted frontier models. The only honest answer is: test it on your workload. If it passes, the privacy win is worth staying local; if it doesn’t, don’t force it.
• Those 12B benchmark numbers. Still not in Google’s official card. I’m treating them as rumor until they are.
• Memory headroom. “26B on 16GB” is real but tight; my own 31B experience is the proof that overhead is not optional in the math.
• The EU AI Act timeline. As of today the high-risk obligations still kick in Aug 2, 2026 in law. But on May 7, 2026 the Council and Parliament reached a provisional political agreement (the “Digital Omnibus”) that would push standalone high-risk obligations to Dec 2, 2027 and embedded ones to Aug 2, 2028. It isn’t adopted yet, so treat the dates as moving. Verify before you rely on them.

That’s where I’ve landed today. The headline I’ll remember: capable AI now runs on my laptop, the compliance story for sensitive data is real, and “local” is a different threat model, not a solved one. That’s where I’ve landed after a week of hands-on testing. I’ll revise as I learn more. If you’ve gone deeper on Gemma’s QAT internals than I have, I’d genuinely like to compare notes.