You want to run powerful AI directly in your business and are wondering how much GPU power you need. The key question is: How many GPUs work together in your server? Because the more cards that compute in parallel, the larger the models and the more concurrent users you can support. We distinguish four configurations: S1 (1 GPU), S2 (2 GPUs), M4 (4 GPUs), and M8 (8 GPUs). What these tiers can actually do only becomes clear on closer inspection.
In this article we compare four concrete server setups for professional on-premise operation. All are based on the NVIDIA RTX Pro 6000 Blackwell Server Edition. Thanks to the Blackwell architecture and 96 GB VRAM per GPU, this card offers the best price-performance ratio for LLM inference in the enterprise server segment. A key advantage: the GPU is modular. A server with 4 cards (M4) can be upgraded to 8 cards (M8) at any time without replacing existing hardware. Depending on the configuration, 1 to 8 cards are used in a single server. You will learn how many tokens per second each configuration generates, how many employees can work with it simultaneously, and which server size fits which tasks. Our data comes from measured throughput values with current open-source models including DeepSeek V4 Flash and GLM 5.2.
At a Glance: Which Configuration for Which Task
Choosing the right server depends primarily on what your AI needs to do in your business. Here is the short version:
- S1 (1 GPU, 96 GB VRAM): The right choice for office work. Models like Qwen 3.6 reliably handle text tasks, summarization, research, and creative work. For daily office work, a single-GPU server is perfectly sufficient.
- S2 (2 GPUs, 192 GB VRAM): The entry point for agentic coding. With double the VRAM and models like DeepSeek V4 Flash, multi-step development workflows become possible: writing, testing, debugging, and optimizing code in a single pass.
- M4 (4 GPUs, 384 GB VRAM) and M8 (8 GPUs, 768 GB VRAM): For frontier models that compete with Claude Opus. Models like GLM 5.2 rank among the most powerful open-source models worldwide. They do not fit into the VRAM of individual cards and are only practical starting at 4 GPUs. For productive teams with multiple users, 8 GPUs are recommended.
The detailed performance data behind these recommendations is explained in the following sections.
Tokens Per Second: What This Number Means for Your Daily Work
Before we dive into server comparisons, let us clarify the most important metric: tokens per second. A token is the smallest unit a language model processes, roughly a syllable or half a word. In English, 100 tokens correspond to about 75 words. So when a server generates 200 tokens per second, that yields roughly 150 words per second, or about 9,000 words per minute. That is far more than a human can read.
For everyday office use this means: raw generation speed is rarely the bottleneck. Much more critical is the server’s ability to handle many requests simultaneously. A KV cache (key-value cache) is the working memory the model needs to avoid recomputing the entire conversation history on every step. Modern inference engines like vLLM use Automatic Prefix Caching to reuse identical prefixes across requests and save memory. The larger the KV cache, the more concurrent conversations or longer context windows are possible.
The following table shows measured throughput values for our four server configurations with different open-source models. All measurements are based on NVIDIA RTX Pro 6000 Blackwell Server Edition GPUs with 96 GB VRAM each.
| Configuration | GPUs | Total VRAM | Model | Generation Single (tok/s) | Generation Concurrent (tok/s) | Input Single (tok/s) | Input Concurrent (tok/s) | Pages per Hour | KV Cache in Tokens | Concurrent Users Office Work | Concurrent Users Agentic Coding |
|---|---|---|---|---|---|---|---|---|---|---|---|
| S1 | 1 | 96 GB | Qwen 3.6 35B A3B | 260 | 900 | 11,000 | 17,000 | 102,000 | 164,000 | 5 | 0 |
| S2 | 2 | 192 GB | DeepSeek V4 Flash | 200 | 1,300 | 6,000 | 10,000 | 60,000 | 1,400,000 | 44 | 10 |
| M4 | 4 | 384 GB | GLM 5.2 | 100 | 400 | 3,000 | 5,000 | 30,000 | 400,000 | 13 | 3 |
| M8 | 8 | 768 GB | GLM 5.2 | 100 | 800 | 4,000 | 10,000 | 60,000 | 4,000,000 | 125 | 31 |
Pages per hour calculation: 400 words per page, 1.5 tokens per word, based on concurrent input throughput. All measurements with NVIDIA RTX Pro 6000 Blackwell Server Edition.
The S2 with DeepSeek V4 Flash reaches 1,300 tok/s concurrently and offers a context window of one million tokens. The S1 with Qwen 3.6 35B A3B achieves 900 tok/s concurrently as an MoE model that parallelizes extremely well with only 3 billion active parameters.
Why a Single-GPU Server Is Enough for the Office and When You Need More
The S1 configuration with a single GPU and 96 GB VRAM is perfectly adequate for many office tasks. With models like Qwen 3.6 it reliably handles text work, summarization, research, and creative tasks. The measured 102,000 pages per hour of input processing means even large document sets can be analyzed in seconds.
But there is one important limitation: the S1’s KV cache is too small for agentic coding. What does that mean in practice? When an AI independently writes, tests, and debugs code, it needs a long conversation history, often tens of thousands of tokens. Each of these tokens occupies space in the KV cache. At 96 GB VRAM, the limit is around 164,000 tokens. That suffices for simple code questions, but not for multi-step programming projects.
Here are concrete recommendations for the S1 configuration:
- Office work and text processing: Excellent. Models like Qwen 3.6 deliver responses in fractions of a second.
- Document analysis and research: Very good. Up to 100,000 pages of input per hour are possible.
- Creative tasks and marketing: Well suited, as medium context windows are entirely sufficient.
- Agentic coding: Not suitable. The KV cache is too small for the necessary context windows.
Qwen 3.6 35B A3B is particularly well suited for the S1 because as a Mixture-of-Experts model it has only 3 billion active parameters. This means: although the model has 35 billion total parameters, only 3 billion are used per computation step. This saves memory and dramatically accelerates processing. Current versions also support a native context of 262,144 tokens.
The S2 with two GPUs and 192 GB VRAM is the next logical step. With DeepSeek V4 Flash, a model with 284 billion parameters, 13 billion of which are active, it achieves a one-million-token context length. This makes agentic coding with multi-step workflows possible. The rule of thumb: for office work an S1 suffices; for demanding AI-assisted code development you need an S2.
From 4 to 8 GPUs: When the Jump to M4 and M8 Pays Off
The M4 and M8 configurations are built around GLM 5.2, a state-of-the-art model from Zhipu AI released under the MIT license. GLM 5.2 ranks among the most powerful open-source models worldwide and competes with Claude Opus on coding benchmarks like SWE-bench Pro. Such frontier models are only practical starting at 4 GPUs (M4); for productive teams, 8 GPUs (M8) are recommended. With a solid one-million-token context and special optimization for long-running tasks, GLM 5.2 is designed for demanding projects.
The comparison between M4 and M8 is revealing: both run the same model and achieve the same single-generation rate of 100 tokens per second. The difference lies in parallel processing. The M8 can handle more requests simultaneously because it has more KV cache memory available.
The table above also shows the number of concurrent users. The M8 is the only configuration truly designed for large teams. With 125 concurrent users for office work or 31 users for agentic coding, it scales far beyond the other configurations. Moreover, the M8 is the first configuration that runs frontier models like GLM 5.2 at a speed usable for teams. The M4, by contrast, is better suited for smaller teams or getting started with frontier models.
The KV Cache: The Invisible Limit of Your AI Infrastructure
Many companies focus exclusively on compute power in TFLOPS or the number of parameters a model can handle when choosing a server. In practice, the KV cache is often the tighter bottleneck, and for the vast majority of applications.
The KV cache (key-value cache) is the memory the model needs to “keep in mind” the history of a conversation or computation. Think of it like a person’s short-term memory. If you give an AI a task with 50 pages of context, it must hold all those pages in the KV cache to reference them later. When the cache is full, the AI can either serve no additional users or must forget older information.
The vLLM documentation describes the KV cache as the central optimization target of modern inference engines. The technique of Automatic Prefix Caching (APC) allows identical beginnings of requests to be reused, saving memory. In practice this means: when multiple employees ask similar questions or access the same documents, the server can share the KV cache and needs less memory.
For businesses in regulated industries such as private banking, fiduciary services, or medical technology, the KV cache is also relevant for data protection. The larger the cache, the more context can be included in a request and the more complex data analyses are possible without data leaving the company. Under the revised Swiss Federal Act on Data Protection (nDSG), personal data must be adequately protected. A sufficiently dimensioned KV cache is a prerequisite for performing complex analyses locally.
The Role of the Context Window: Why One Million Tokens Is More Than Just a Number
The models DeepSeek V4 Flash, GLM 5.2, and Qwen 3.6 all support context windows of one million tokens or more. That sounds abstract, but it has very concrete effects on your daily work.
A context window of one million tokens means: you can feed the AI your most important product manual, the last fifty customer inquiries, the current proposal template, and even the team meeting minutes, all in a single request. The AI keeps track of all this information and can identify connections that would be lost across multiple separate requests.
The DeepSeek V4 Flash model uses a Hybrid Attention architecture specifically optimized for long contexts. Unlike earlier models, DeepSeek does not need to compare every position in the text with every other position but combines local attention (for immediate context) with compressed, far-reaching attention (for overall context). This saves compute power and enables real-time processing of one million tokens.
GLM 5.2 from Zhipu AI uses a similar optimization. The model employs a technique called IndexShare, where multiple attention layers share the same index. This reduces compute cost at one million tokens by a factor of 2.9. GLM 5.2 is released under the MIT license and achieves 62.1 percent on coding benchmarks like SWE-bench Pro. That is on par with Claude Opus 4.8 (69.2 percent). However, a model of this size requires a corresponding number of GPUs: GLM 5.2 does not fit into the VRAM of a single card, it only becomes usable starting at 4 GPUs, and for productive use with multiple users, 8 GPUs are recommended. Both models show that architecture is often more important in practice than raw parameter count.
Power and Cooling: What the Servers Need in Operation
One aspect often overlooked when choosing a server is power draw. An AI server typically runs around the clock and consumes substantial amounts of energy. Power consumption increases with the number of GPUs and the associated cooling.
The S1 configuration with one GPU draws about 0.8 kilowatts. That is roughly equivalent to a gaming PC. Heat output is moderate, so the server can operate in most server rooms without special air conditioning.
The S2 with two GPUs rises to about 1.3 kilowatts. Cooling becomes more demanding here. A server room or at least a well-ventilated equipment room is recommended. The heat output is roughly equivalent to a small space heater.
The M4 and M8 configurations with four and eight GPUs respectively reach 3 to 5 kilowatts. These systems require a professional server room with climate control. The waste heat is substantial and must be actively removed. In practice, this means an M8 server needs its own room with appropriate infrastructure.
When planning your AI infrastructure it is important to consider these boundary conditions early. A server that can be placed in a basement or storage closet significantly reduces planning effort. A system that requires its own server room demands corresponding building modifications.
FAQ
What is the difference between a token and a word? A token is the smallest processing unit of a language model. In English, about 100 tokens correspond to 75 words. Tokens are not whole words but can be syllables, characters, or word parts.
How many concurrent users can an S1 server handle? For pure office work with small context windows, about 5 concurrent users. For agentic coding the S1 is not suitable because the KV cache is too small for the required context lengths.
Do I absolutely need a server with multiple GPUs for AI in the office? No. An S1 with a single GPU and models like Qwen 3.6 is perfectly sufficient for most office tasks. Multiple GPUs only become interesting when many employees work simultaneously or agentic workflows are used.
What does agentic coding mean exactly? Agentic coding means the AI not only answers individual code questions but independently handles multi-step tasks: writing code, testing, debugging, and optimizing. This requires very large context windows and correspondingly large KV cache.
The RTX 6000 Pro Blackwell does not have NVLink. Is that a problem? No, it is not a problem. Modern architectures like Mixture-of-Experts models distribute their experts directly across GPUs, so only minimal routing information needs to be exchanged between cards per computation step. This data volume is small, and the critical factor is latency, not bandwidth. PCIe 5.0 with its low latency is entirely sufficient. NVLink would primarily benefit dense models with large weight exchanges, but these are no longer the first choice for LLM inference in practice.
Can I run different models on the same server simultaneously? Yes, modern inference engines like vLLM allow parallel operation of multiple models. However, available resources (VRAM, KV cache) are then shared, which reduces the maximum number of concurrent users.
Why is the KV cache more important than raw compute power? Because in practice most AI applications are limited by memory bandwidth, not raw compute. A generously dimensioned KV cache allows more concurrent users and longer contexts, which directly impacts productivity.
Which model is best for a small or medium-sized business? It depends on the specific tasks. For general office work, Qwen 3.6 is excellent since it runs extremely efficiently with only 3 billion active parameters. For demanding coding tasks and agentic workflows, we recommend DeepSeek V4 Flash or GLM 5.2. Both are available under open licenses (MIT) and optimized for professional use.