← Retour au fil
Chutes Pushes Single-GPU Inference to Nearly 18,000 Tokens per Second
TAO Daily12 nov., 17h · il y a 4j

Chutes Pushes Single-GPU Inference to Nearly 18,000 Tokens per Second

In an early inference test, Chutes (SN64) ran its Parallax AI model on a single Nvidia RTX 5090, generating nearly 18,000 tokens per second across 256 concurrent requests. […]

Bittensor

Détails

Source
TAO Daily
Publication
12 nov. à 17h04

Contenu source (brut)

<p class="wp-block-paragraph"><a href="https://taodaily.io/chutes-sn64-parallax-built-a-global-ai-supercomputer-from-240-consumer-gpus/">Parallax</a> reached nearly 18,000 tokens per second on a single Nvidia GeForce RTX 5090 while handling 256 concurrent requests. The figure reflects the combined output across all requests. <a href="https://taodaily.io/chutes-opens-6-12-billion-ai-requests-for-harvard-research-to-help-build-better-ai/">Chutes (SN64)</a> recorded it in an early performance test using a custom version of vLLM that the team is still tuning.</p> <p class="wp-block-paragraph"><a href="https://taodaily.io/chutes-sn64-becomes-launch-partner-for-tencents-new-hy4-ai-model/">Chutes (SN64)</a> provides on-demand access to AI models through a distributed network of GPUs. Its <a href="https://taodaily.io/parallax-made-it-possible-to-train-giant-ai-models-on-scattered-ordinary-gpus/">Parallax</a> model has 8 billion parameters but activates only about 1.25 billion to process each token. Its expert layers use 4-bit arithmetic, and most of its attention layers use a fixed-size cache. These choices reduce computation and memory demands, giving the GPU room to handle more requests at once.</p> <figure class="wp-block-image size-large is-resized"><img loading="lazy" decoding="async" width="1024" height="412" src="https://taodaily.io/wp-content/uploads/2026/11/image-100-1024x412.png" alt="" class="wp-image-26317" style="aspect-ratio:2.4860557768924303;width:624px;height:auto" srcset="https://taodaily.io/wp-content/uploads/2026/11/image-100-1024x412.png 1024w, https://taodaily.io/wp-content/uploads/2026/11/image-100-300x121.png 300w, https://taodaily.io/wp-content/uploads/2026/11/image-100-766x308.png 766w, https://taodaily.io/wp-content/uploads/2026/11/image-100.png 1903w" sizes="(max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption"><a href="https://chutes.ai/">Chutes’ Website</a></figcaption></figure> <p class="wp-block-paragraph">For inference providers, generating more tokens per second on the same hardware can bring down the cost of serving each response. The preview measures how much output the card can produce under heavy load. Metrics such as time to first token and generation speed per request would show how quickly individual users receive their answers.</p> <p class="wp-block-paragraph">Chutes also designed Parallax’s training around consumer hardware. The team pre-trained the model across hundreds of GPUs in multiple countries, including fleets of RTX 5090s. Each group of GPUs trained a subset of the model’s experts and used lightweight approximations for the others. This reduced the amount of data the groups needed to exchange and made it possible to train over ordinary network connections.</p> <p class="wp-block-paragraph">Chutes expects further tuning of its vLLM fork to increase throughput. The early result shows how far the team has already taken a model designed for widely available hardware, from training it across hundreds of consumer GPUs to serving 256 simultaneous requests on a single card.</p> <p class="wp-block-paragraph">➛ Access Parallax <a href="https://storage.googleapis.com/chutes-random/parallax-draft-tech-report.pdf">Here</a>.</p> <figure class="wp-block-embed is-type-wp-embed is-provider-the-tao-daily wp-block-embed-the-tao-daily"><div class="wp-block-embed__wrapper"> <blockquote class="wp-embedded-content" data-secret="40oNbeAw05"><a href="https://taodaily.io/chutes-parallax-shows-distributed-gpus-can-train-recurrent-models-without-blocking/">Chutes&#8217; Parallax Shows Distributed GPUs Can Train Recurrent Models Without Blocking</a></blockquote><iframe class="wp-embedded-content" sandbox="allow-scripts" security="restricted" style="position: absolute; visibility: hidden;" title="“Chutes’ Parallax Shows Distributed GPUs Can Train Recurrent Models Without Blocking” — The TAO Daily" src="https://taodaily.io/chutes-parallax-shows-distributed-gpus-can-train-recurrent-models-without-blocking/embed/#?secret=IVw0jq9zKF#?secret=40oNbeAw05" data-secret="40oNbeAw05" width="500" height="282" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe> </div></figure> <p class="wp-block-paragraph"></p>