Open-Source Inference Just Had Its Wildest Month Ever

There are months where the open-source inference world moves at a normal pace, and then there’s the first week of August 2026. In seven days, we saw a 284-billion-parameter model running on a MacBook, a 2.4-trillion-parameter MoE drop from Alibaba, and FP8 KV caches quietly becoming the new production default. If you blinked, you missed a paradigm shift. Let’s start with the biggest story nobody is talking enough about.

For years, the KV cache—that invisible structure that stores attention keys and values for every token a model has processed—has been the silent killer of long-context inference. Stored in 16-bit precision by default, the KV cache often fills GPU memory long before the model weights do. Developers would buy 80GB GPUs and then watch context windows die at 30,000 tokens because the cache ate everything. FP8 KV caching changed that math in August 2026. It went from a research curiosity to a production default in what felt like days. The most compelling data comes from Cloudflare, which published a detailed technical breakdown of running Kimi K2.6 and GLM 5.2 on Workers AI with FP8 KV cache quantization. The numbers tell a story that GPUs are finally being used the way they should be: | Concurrent Requests | BF16 KV Cache (tok/s) | FP8 KV Cache (tok/s) | |---|---|---| | 1 | 137 | 125 | | 8 | 731 | 689 | | 16 | 1,106 | 1,028 | | 32 | 1,558 | 1,489 | | 64 | Out of Memory | 2,192 | For Kimi K2.6, the FP8 KV cache effectively doubles available context—from roughly 686,000 tokens to about 1.37 million on a single H200. There’s a small per-token throughput cost at any given concurrency level, but that’s irrelevant when the BF16 cache runs out of memory at 32 concurrent requests and FP8 keeps going to 64. Peak throughput jumps 41 percent higher than BF16’s ceiling, and cost per token drops about 30 percent. The common objection to FP8 has always been accuracy. Cloudflare’s testing across GSM8K, MMLU, ARC-Easy, ARC-Challenge, and internal tool-calling benchmarks showed FP8 and BF16 caches are effectively indistinguishable—differences stay within measurement noise. One Reddit user on r/LocalLLaMA summed up the community mood: “FP8 KV cache is the best default on Hopper/Blackwell—2× capacity, no measurable accuracy loss, no throughput hit. Reach for --kv-cache-dtype fp8 first.” It’s not just Cloudflare. NVIDIA’s official documentation now recommends FP8 KV cache as the default, noting roughly 50 percent memory reduction and 9–17 percent faster context attention on Blackwell for d=128 models. SGLang—the framework Cloudflare uses in production—now ships robust FP8 KV cache support across multiple attention backends. The vLLM project merged PR #34741 enabling FP8 KV-cache for AMD RDNA4 via software dequantization, doubling KV-cache capacity for a Qwen3-Coder-30B AWQ model on an AMD Radeon AI PRO R9700. A Hacker News comment captured the long-context implications: “The Cloudflare numbers are wild. 1.37M context on a single H200? That changes what's possible with long-context agents.”

80B Models in 4.3GB: The MoE Streaming Revolution

If FP8 is about making GPUs more efficient, the other big August story is about making GPUs optional. Swiftlet, a Swift + Metal runtime released in early August, runs Qwen3-Next-80B-A3B—an 80-billion-parameter MoE model—in just 4.3GB of peak RAM on Apple Silicon Macs. For context, that’s less memory than a modern web browser uses with thirty tabs open. | Model | Disk | Peak RAM | Decode Speed (M5 Mac) | |---|---|---|---| | Qwen3.6-35B-A3B (4-bit) | 18 GB | 2.6 GB | 7–11 tok/s | | Qwen3-Next-80B-A3B (4-bit) | 42 GB | 4.3 GB | 4.5–5 tok/s | The trick isn’t magic; it’s exploiting MoE sparsity properly. MoE models activate only about 3 billion parameters per token. Swiftlet keeps the dense core—attention, router, shared experts—in RAM and streams expert weights from SSD on demand. As the project’s README puts it: “The model never has to fit in RAM—only its working set does.” The same approach powers the 35B version running natively on an iPhone 17 in about 2.5GB of RAM at roughly 1 tok/s. The developer noted: “As far as we know, that is the first time a model of this class has run natively on a phone.” A Hacker News user with 32GB of RAM on their MacBook wondered aloud why the runtime couldn’t use extra RAM to go faster—a fair question, and likely a roadmap item. Mference, another Swift + Metal inference engine, takes the same philosophy further. It runs DeepSeek-V4-Flash—a 284B-parameter MoE with 13B active per token and a 1M-token context window—in about 6.8GB of peak RAM on a 24GB Mac, hitting up to 4.8 tok/s. The model was officially open-sourced on July 31 under an MIT license. Mference streams its 256 experts from SSD with “shared-KV MQA attention with compressed long-range KV.” The Hacker News reaction to that was immediate: “284B model on a 24GB Mac at 4.8 tok/s? This is what happens when you actually exploit MoE sparsity instead of pretending it's a dense model.” Swiftlet and Mference aren’t alone. August brought a wave of expert-streaming runtimes: KTransformers keeps attention on GPU while offloading MoE experts to system RAM for 671B-class models; oMLX arrived as a new LLM inference server with continuous batching and SSD caching; PocketMoE targets 300B-class models on consumer GPUs; and a llama.cpp fork called Expert Streaming Engine runs GPT-OSS 120B F16 at 140 t/s prefill and 11.5 t/s decode on 8+10GB Ampere hardware. The architectural insight here runs deep. A developer covering Swiftlet for the Chinese AI press noted that the entire approach works because “each token only activates a small portion of expert parameters—an 80B model actually only computes with about 3B parameters per token.” That’s not a workaround. That’s the design.

The Model Releases That Changed the Game

None of this inference progress matters without models worth running. August 2026 delivered. DeepSeek officially released DeepSeek-V4-Flash on July 31 under the MIT license. The 284B MoE (13B active) model features 1M context and scored 82.7 on Terminal Bench 2.1 and 54.4 on DeepSWE—substantially stronger than the preview version. The killer stat for production deployments: a concurrent limit of 2,500, five times that of V4-Pro. The pricing structure is aggressive. In China, input with cache hits runs 0.02 yuan per million tokens during off-peak hours; cache misses cost 1 yuan and output runs 2 yuan per million tokens. International pricing sits at $0.14 per million tokens for cache-missed input and $0.28 per million for output. Compare that to GPT-5.6 Sol pricing at $5 input / $30 output per million tokens, and the gap is somewhere between tens and hundreds of times. Early developer feedback suggests the quality holds up. One Chinese AI developer tested V4-Flash against GPT-5.6 Sol + Codex on the same task: roughly 40 seconds versus 1 minute 47 seconds. Another developer reported that V4-Flash beats GLM 5.2 among domestic models but is weaker than GPT-5.6 in some head-to-head tests, with tool-calling accuracy generally solid. Alibaba answered with Qwen3.8-Max on August 3—a 2.4-trillion-parameter MoE model with roughly 95B active parameters, 1M context, and native multimodal vision. Reported inference cost is about one-eighth of industry standards. Alibaba’s Zhēnwǔ M890 supernode was adapted alongside, delivering up to 1.5× performance improvement in agentic reasoning scenarios. Huawei open-sourced openPangu-2.0-Pro: 505B total parameters, 18B sparse activation, 512K context, trained on 34T tokens. The headline claim: it’s the first 500B-plus frontier model completely trained on non-NVIDIA hardware. Deep adaptation for Ascend NPUs yields roughly 2× single-card throughput versus mainstream open-source models, and a native 30B terminal-side model claims 50 percent faster inference with 20 percent less memory usage. MiniMax also open-sourced H3, its first multimodal generation model supporting text, images, audio, and video with 2K resolution and 15-second native audio-video output. One caveat for commercial users: the MiniMax Community License excludes the US, EU, UK, and South Korea—a pointed reminder that “open source” in 2026 often comes with geopolitical strings attached.

The Quantization Frontier Goes Deeper

FP8 KV cache grabbed the headlines, but the quantization research kept pushing beyond it. Cloudflare’s INT4 weight compression for GLM 5.2 shrinks checkpoints from 705GB to 421GB—a 40 percent reduction. In an eight-way tensor-parallel setup, per-GPU memory drops from 88GB to 52GB. Low-concurrency decode speed jumps 55 percent from 60 tok/s to 92 tok/s. The why is straightforward: decode is memory-bandwidth-bound, and less data to move means faster tokens. Accuracy holds: GSM8K scores 94.39 percent (INT4) versus 93.56 percent (FP8), with MMLU within 0.1 points. TurboQuant, the first open-source implementation of Google’s ICLR 2026 paper, delivers near-optimal KV cache compression at 5× with near-zero quality loss. The turboquant-plus-vllm package offers 4.3× weight compression plus 3.7× KV cache with zero calibration. The market reaction was chaotic: some investors misread the paper as a threat to storage hardware, triggering panic selling that massively devalued related stocks. A separate group of researchers claims the paper has technical issues that were flagged before submission—an unresolved controversy worth watching. FP4 KV cache is already in the labs. NVIDIA’s own testing shows less than 1 percent accuracy loss versus BF16 and FP8 across LiveCodeBench, MMLU-PRO, MBPP, and Ruler 64K, with first-token latency improvements of up to 3×. SGLang reports FP4 performs well on large models like Qwen3-235B-A22B but shows more noticeable accuracy drops on smaller models like gpt-oss-120b. Production users remain cautious: one NVIDIA developer forum thread called basic q4_0 KV cache “unusable at scale,” noting prompt processing crashes on 64K contexts due to software dequantization overhead. FP4 is the future; FP8 is the present.

Speculative Decoding Without the Draft Model

Packed twin inference brings a clever twist to speculative decoding: up to 2× LLM throughput using free speculative drafts from n-gram lookup and the model’s own MTP head, all verified in a single packed decode. No draft model, byte-identical output. The README explains: “Wrong guesses are discarded and recomputed—output never changes, only the number of passes does.” Dell holds related patents on twin inference dating back to 2023, and an open-source implementation with HIP kernels computes two matmul accumulations in a single uint16 load. It’s early, but the throughput-per-dollar math is compelling. An arXiv paper from August 3 introduces Bole for efficient tree speculation in hybrid-attention language models. Meanwhile, optillm—an open-source inference proxy—reached maturity in July with full local inference support and a plugin architecture. The community is also eyeing new runtimes. One Hacker News user called Valkyr “a fresh take on LM Inference runtimes. It's quite different from llama.cpp, vLLM, or ZINC.” Another announced: “We're enabling the first cross-platform in-app inference experience for GGUF models and we're soon shipping our own inference kernels fully optimized for mobile.”

The Hidden Cost of Squeezing: Cache Integrity

When you pack more requests onto the same GPU with FP8 and INT4, you create a new attack surface: hundreds of requests sharing the same physical KV cache pages. A memory corruption bug could leak tokens from one request to another. Cloudflare added KV cache integrity checking as a defense layer. Each physical cache page gets a tag updated on reallocation. Before decode operations read the cache, mappings are verified; mismatches abort the request rather than return corrupted data. The overhead is nearly invisible—throughput falls less than 1 percent, and tail latency stays within 1 percent. It’s the kind of unglamorous engineering that makes the headline numbers safe to deploy. Cloudflare plans to extend FP8 KV cache to more clusters, test NVFP4 weights on Blackwell, and reduce the cost of persistent integrity checks. Their partnership with NVIDIA already brings GPU-accelerated inference to Cloudflare’s edge network through AI Gateway 2.0.

The Integrity of the Numbers

As with any fast-moving sector, verified facts and community claims are mixing in strange ways. It’s worth separating what we know from what’s still unclear. The FP8 KV cache results are production data from Cloudflare and confirmed in NVIDIA documentation. The Swiftlet and Mference RAM numbers come directly from project documentation and are reproducible. The DeepSeek-V4-Flash pricing is public API data. The Qwen3.8-Max cost claims are harder to verify—Alibaba has disclosed model architecture and performance numbers but has not published training cost specifics. Community estimates of training cost exist, but they’re estimates.

What This Means for the Industry

The market context for all this is staggering. TrendForce projects AI server shipments will grow over 28 percent in 2026, with inference workloads increasingly driving demand. Huang expects the AI inference chip market to reach a potential $1 trillion by 2027. China’s Academy of Information and Communications Technology reports daily token calls in China grew 1,400-fold in two years, surpassing 140 trillion tokens in early 2026, with inference compute growing 10,000-fold in the same period. The inference market in China is expected to double to about 87.65 billion yuan. Meanwhile, the cost of running frontier-class models keeps collapsing. One analysis shows the cost of reaching early-2025 high-end reasoning capability dropped 128-fold. GPT-4-level inference costs fell from roughly $20 per million tokens in late 2022 to around $0.40 per million tokens now. The tension: IT leaders’ top cloud concern shifted from security to cost in 2026, according to Broadcom research—31 percent of leaders now cite cost as their primary concern, up from 26 percent in 2025. And even as token prices fall, total AI spend isn’t necessarily dropping. Agent workloads consume 10–100× more tokens per task than single prompts, so organizations may end up spending more even as unit costs plummet.

The Bottom Line for Practitioners

The practical takeaways from August 2026 are clear enough: Start every new production deployment with --kv-cache-dtype fp8. It’s the new default for a reason, and Cloudflare’s production data validates it. For edge and consumer devices, Swiftlet and Mference prove MoE sparsity is an inference superpower, not just a training trick. The ability to run 80B-class models in 4.3GB RAM changes what “local AI” means. For AMD users, vLLM’s RDNA4 FP8 support is now production-ready. The 284B DeepSeek-V4-Flash runs on a 24GB Mac. There are still rough edges on RDNA4—community reports cite FP8 GEMM kernel regressions and power-limit weirdness in some configurations—but the direction is unmistakable. Long-context AI agents are no longer constrained by memory. 1.37M tokens on a single H200 was science fiction in 2025. It’s a supported configuration now. The open-source inference stack has never been this powerful—or this cheap. The question isn’t whether these technologies will spread. It’s which of these projects will survive contact with the enterprise, and which clever GitHub experiments will quietly fade into the commit history. For now, the momentum favors the ones with production numbers, and August 2026 delivered plenty of those.