700 Tokens Per Second: Google TPU Performance Benchmark Shakes AI Chip Market

Google’s custom AI silicon has reached a critical speed milestone, with benchmarks showing its seventh-generation TPUs capable of generating 700 tokens per second for a single user. This performance figure, achieved while serving a massive open-weight model, marks a significant technical leap that challenges the prevailing dominance of Nvidia hardware in large-scale AI inference.

The breakthrough is driven by the release of new, PyTorch-native versions of two major inference engines: SGLang and vLLM. Previously, running these tools on Google’s Tensor Processing Units required using the JAX framework, a system many developers found less familiar than PyTorch. The new implementations allow the hardware to operate natively within the PyTorch ecosystem, largely eliminating the need for code rewrites.

This shift is not merely theoretical; it is already operational. Meta, the company behind the PyTorch framework, confirmed that the new SGLang-TorchTPU engine is currently running in production. Zongwei Zhou, Meta’s PyTorch-TPU lead, described the result as a "remarkably mature, PyTorch-native serving stack" that emerged from a three-way collaboration between Meta, Google, and the SGLang team.

Meta’s adoption follows a period of hesitation. The company previously used an older bridge called PyTorch/XLA to access TPUs between 2020 and 2023 but eventually canceled those contracts. Jana van Greunen, director of PyTorch engineering at Meta, stated that the company "cannot be constrained by a single hardware architecture." The new TorchTPU stack enables Meta to deploy its workloads to Google hardware without migrating to a different software ecosystem.

Commercially, this marks the first time Google has sold its TPUs to an external customer for production use. While the exact location of Meta’s production units is not specified, reports indicate that talks last year included Google Cloud rentals for 2026 and purchases for 2027. Meta is also reportedly set to sign a multi-year rental agreement, solidifying a long-term partnership.

The move appears to be a direct counter to Nvidia’s software lock-in. Just one day before Google’s announcement, Amazon revealed it was open-sourcing TorchNeuron, a similar native PyTorch backend for its own Trainium chips. Both companies are actively working to decouple AI software from specific hardware vendors, giving enterprises more flexibility in their infrastructure choices.

Performance comparisons highlight the competitive gap. A tuned vLLM setup on 16 Google Ironwood chips outperformed the same number of Nvidia GB200 chips, which reached 452 tokens per second. Without speculative decoding, the new kernels for specific models deliver 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes.

Looking ahead, the ecosystem is expanding rapidly. SGLang-TorchTPU will begin supporting DeepSeek V4.1 Flash and Reflection Beam immediately. RadixArk, the startup behind SGLang, aims to provide Day 0 support for new open models by the first quarter of 2027. Google, Meta, and RadixArk will share further details at the SGLang Summit in San Francisco on November 12 and 13.

For New England tech hubs and enterprises relying on AI infrastructure, this development offers a tangible alternative. The ability to run high-performance inference on non-Nvidia hardware could influence future data center planning and procurement strategies in the region.