Can You Fit a 70B Model on a Single RTX 5090? Google’s TurboQuant Says Yes

The AI industry loves a big number. Trillion-parameter models. Million-token context windows. Massive GPU clusters that cost more than most houses. But some of the most important work happening right now has nothing to do with scale. It’s about compression. Basically: doing more with less. And a quiet Google Research paper, nearly a year old, is about to get its moment in the spotlight.

TurboQuant is a compression algorithm that reduces the memory footprint of large language models by up to 6x. Zero accuracy loss. No retraining required. The paper first appeared on arXiv in April 2025, but Google is featuring it on their Research blog along with some experiments and results this week ahead of its formal presentation at ICLR 2026 in late April.

So why should you care about a year-old research paper? Because it attacks a problem that everyone running AI bumps into eventually: the key-value cache bottleneck.

What’s the KV Cache Problem?

Here’s the short version. When you chat with an LLM, the model doesn’t just process your latest message. It keeps a running record of the entire conversation in something called the key-value (KV) cache. Think of it as the model’s short-term memory for your session. A sort of journal tracking all the things that matter in the conversation you’re having with your LLM of choice.

The problem: that memory grows with every turn (prompt/response). The longer the conversation, the more GPU memory it consumes. For long-context tasks like document analysis, code reviews, or multi-step research, the KV cache can balloon to the point where it pushes out the model itself.

If you’ve ever had a long conversation with ChatGPT, Claude, or Gemini and noticed things slowing down, or gotten a message that your context is too long, that’s the KV cache hitting its limit. We’ve all experienced compaction (compacting), which is another method to manage the cache by throwing away what you don’t need in a conversation. Cloud providers manage this behind the scenes with massive hardware. But the constraint is real, and it costs them (and eventually you) money. For anyone running models locally, on a single GPU in a lab or small office, there’s no hiding from it.

Cloud providers can throw hardware at this. Small labs and small businesses can’t.

The Small Lab Reality Check

We run an AI research lab here at Stark Insider (StarkMind). Our main rig is a Threadripper-based RTX 5090 with 32 GB of VRAM. It handles 35-billion parameter models comfortably for daily work (Qwen 3.5 35B is my current local favorite). But when we fire up our 120B open-source model (GPT OSS 120B still holding strong) for overnight evaluation and research jobs, things get tight fast. And I should also mention being based here in Silicon Valley: things get very hot. I spent far too much time on cooling and air flow management.

The KV cache is usually what kills us; resulting in dreaded OOM (out-of-memory) issues. We can load the model weights fine. But as conversation context grows during long, multi-turn benchmark runs, VRAM fills up. We run an internal eval harness called StarkBench that puts models through multi-turn research synthesis and film recommendation tasks. When we tested gpt-oss:120b and Qwen 3.5 122B this week, we had to dial the num_ctx parameter (the context window size in Ollama) down from 32K to 16K tokens just to avoid those pesky out-of-memory crashes. That’s a direct KV cache constraint I’ve painfully learned. With a 120-billion parameter model spilling over into system RAM, there’s simply not enough VRAM left for a full-size context window. Each run took about 18 minutes with CPU spillover. It works sure. But it’s duct tape, and not what I’d consider an ideal solution.

Some of the most important work happening right now has nothing to do with scale.

On paper, a 6x reduction in KV cache memory would change that equation significantly. Instead of capping num_ctx at 16K, we might fit the full 32K. Instead of one long evaluation running overnight, we could potentially run several. For a small lab like ours (and yours), that’s could make for a meaningful difference.

The catch? TurboQuant is a research paper, not a product as I’ve discovered. You can’t pip install it. It’s not in vLLM, llama.cpp, Ollama, or any of the serving frameworks that AI labs and developers actually use. The paper has been public for nearly a year, and none of those projects have merged it yet.

That’s worth noting I think. Plenty of impressive research never makes it into the tools people use every day. Then again, this is Google so I would expect this to eventually find its way into the mainstream. It’s possibly already being used for future versions of Gemini.

Peer Review and Early Adopters

That said, the research has real credibility behind it. TurboQuant was accepted at ICLR 2026 (April 23-25), one of the most selective machine learning conferences in the world. Its companion papers also passed peer review at top venues:  a quick Google search revealed that QJL was published at AAAI 2025, and PolarQuant was accepted at AISTATS 2026.

And within hours of Google’s blog post going live, independent developers started implementing TurboQuant from scratch. Not using Google’s code, because Google hasn’t released any that I could find. These are people reading the paper and writing their own implementations based on the math alone. Check Reddit and X for the early adopters who are sharing their findings and experiences.

One developer built a PyTorch implementation with a custom Triton kernel, tested it on a Gemma 3 4B model running on an RTX 4090, and got character-identical output to the uncompressed baseline at 2-bit precision (they’ve also made the code available for download if you’re adventurous). Another got it running on Apple Silicon via MLX with a 35B model, scoring 6 out of 6 on needle-in-a-haystack tests at every quantization level. Over in the llama.cpp community, at least three developers are working on C and CUDA implementations, with one reporting 18 out of 18 tests passing and compression ratios matching the paper’s claims.

That’s a good sign. The math is likely reproducible and the results hold up outside Google’s benchmarks.

A few caveats, though I discovered when reading more about the research and announcement. Google’s own experiments only tested on 8-billion parameter models (Gemma, Mistral, Llama 3.1). Whether TurboQuant scales cleanly to larger models is still undemonstrated — though it’s worth noting small models are in vogue these days due to their usability with less compute. The headline “8x speedup” refers specifically to attention computation, not end-to-end inference. And one early implementer found that the QJL error-correction component is tricky to get right. The naive approach produced garbage output. Getting the full pipeline working correctly requires careful adherence to the paper’s asymmetric estimator design.

What the Benchmarks Show

With those caveats noted, the results are hard to ignore. Google tested TurboQuant across standard long-context benchmarks:

• 3-bit quantization of the KV cache with no training, no fine-tuning, and no measurable accuracy loss
• Perfect scores on needle-in-a-haystack tests (finding a single fact buried in massive text) across all benchmarks
• Up to 8x speedup in computing attention on H100 GPUs compared to unquantized 32-bit keys
• Superior recall ratios in vector search compared to state-of-the-art methods, even those using larger codebooks and dataset-specific tuning

That last point matters for anyone building search. TurboQuant isn’t just about chat. It also speeds up vector search, the technology behind semantic search engines and RAG (retrieval-augmented generation) pipelines. Faster index building with near-zero preprocessing time. Basically, lower memory and better recall. For search infrastructure, that’s a meaningful combination. For StarkMind, I’ll be looking to pilot TurboQuant with some of our RAG pipelines that we experimented with last year.

Bigger Than Bigger

Here’s what I think is the real story, and it has nothing to do with TurboQuant specifically.

The AI conversation for the past few years has been dominated by scale. Bigger models. More parameters. Larger context windows. Every morning we wake up to a new Frontier LLM or local model that promises X% improvements across all the usual synthetic benchmarks. And yes, scale matters of course. But the most consequential breakthroughs might not come from building the next trillion-parameter behemoth. They might come from clever tricks like this. Compression. Quantization. Efficient math.

Because compression is what puts AI in places it can’t go today. Edge devices. Phones. Embedded systems. A medical clinic with a single workstation. A law firm that needs to keep client data on-premises. A startup that can’t afford a $50,000/month cloud GPU bill. Our small lab here at Stark Insider.

The parameter race may make headlines, but compression makes deployment possible to memory challenged edge devices, workstations and even smartphones (check out Enclave on your iPhone!).

The KV cache problem is getting attention from multiple angles. Some workarounds already exist today. Tools like llama.cpp support basic KV cache quantization (q4_0, q8_0). Modern model architectures like Grouped Query Attention reduce KV cache size by design. Sliding window attention caps the cache at a fixed number of recent tokens. Even the compaction you experience I mentioned earlier in ChatGPT or Claude is a form of KV cache management: the system summarizes or drops older context to stay within limits.

TurboQuant’s contribution is pushing quantization to extreme bit-widths (3-bit) without the accuracy loss that existing methods introduce. GGUF quantization has already made it possible to run models locally that would have required a data center two years ago. Techniques like speculative decoding, flash attention, and PagedAttention have all chipped away at the compute problem from different angles. The trend is clear. And it’s accelerating.

What to Watch For

TurboQuant will be formally presented at ICLR 2026 in late April (main conference runs April 23-25). Also, its companion paper, PolarQuant, will be presented at AISTATS 2026 around the same time.

If you’re a developer or running a lab, the thing to watch is whether any of the major open-source serving frameworks merge this. The techniques are described as “exceptionally efficient to implement” with “negligible runtime overhead,” and the early independent implementations suggest that’s true. Whether it becomes a checkbox in Ollama or a flag in llama.cpp is the real question.

For everyone else, the takeaway is simpler. The AI industry is learning that you don’t always need a bigger model, or boat. Sometimes you need a smarter one. Or at least, smarter plumbing.