Run Apertus AI v1.5 70B with vision, tool calling, and up to 229k tokens of context on a single enterprise GPU: RTX 6000 Blackwell

Swiss AI is Switzerland’s national AI initiative: a collaboration of ETH Zurich and EPFL Lausanne, with models trained on the “Alps” supercomputer at the Swiss National Supercomputing Centre (CSCS) in Lugano.

Apertus v1.5 70B has just been released, and it adds what makes a model actually useful in day-to-day work: vision input (charts, screenshots, scanned documents), native tool calling in the structured OpenAI format, and a context window grown from 65k to 262k tokens. The official reference deployment needs two datacenter NVIDIA H200 GPUs, but a quantized version serves 192k tokens comfortably on a single NVIDIA RTX PRO 6000 Blackwell, with a measured maximum of 229k when using nearly the entire 96 GiB card.

Compliance by design

Apertus is designed for transparent, auditable deployment under European regulatory requirements, with open weights, training documentation, data-governance disclosures, and EU AI Act transparency materials. For any organization operating under the EU AI Act, that is a very different proposition from a vendor black box.

Apertus is not only open weights. It is built so that legal and compliance teams can sign off on it. The model is trained exclusively on publicly available data, with websites’ machine-readable opt-outs respected even retroactively. Personal data is filtered out before training, and the pipeline technically suppresses verbatim memorization of training text. Weights, data, recipes, and the transparency documentation required by the EU AI Act are all published under Apache 2.0.

The Catch

It ships as approximately 145 GB (135 GiB) of BF16 weights, and the official deployment reference runs it on two datacenter NVIDIA H200 GPUs: NVLink boards, the whole machine room. That is exactly the kind of dependency on-premise AI is supposed to avoid.

The Solution

We developed a production-proven three-tier mixed-precision quantization that runs Apertus v1.5 70B with a 192k context profile on a single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), the GPU of choice for enterprise on-premise inference servers. At a fraction of the hardware cost, the compression alone makes decoding 55% faster while using 32% less weight storage than the FP8 variant.

TL;DR: Just Tell Me How to Run It

We publish the 48 GiB NVFP4 checkpoint on Hugging Face and the temporary vLLM build recipe on GitHub. Until Apertus 1.5 support lands in a native vLLM release, the matching container is published as onpremai/vllm-apertus-1p5 on Docker Hub. All you need is a machine with an NVIDIA RTX PRO 6000 Blackwell (96 GB), NVIDIA drivers, and Docker with the NVIDIA container toolkit:

docker pull onpremai/vllm-apertus-1p5:latest

docker run --gpus all -v /path/to/model:/model \
  onpremai/vllm-apertus-1p5:latest \
  --model /model \
  --served-model-name apertus-v1.5-70b \
  --host 0.0.0.0 --port 8080 \
  --dtype auto \
  --chat-template /model/chat_template.jinja \
  --tool-call-parser apertus --enable-auto-tool-choice \
  --reasoning-parser apertus

About three minutes later you have an OpenAI-compatible endpoint on port 8080, and you can talk to it with text, tools, or images through the standard chat-completions API:

# 3. Talk to it
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apertus-v1.5-70b",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "What does this chart show?"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
    ]}]
  }'

Any OpenAI-compatible client works unchanged: LangChain, the openai Python SDK, or a gateway like LiteLLM in front of it (see our LLM API standards guide). The rest of this article is the full recipe behind it.

Resources: NVFP4 weights and model card · FP8 reference checkpoint · Docker build source · Docker Hub image

What quantization is, and why it is not a solved button-press

A 70B-parameter model in BF16 stores every weight in 16 bits: 2 bytes x 70 billion, plus embeddings, totalling 135 GiB before a single token of your prompt is processed. Quantization stores those weights in fewer bits (8, or even 4) and rescales them so the matrix multiplications still come out approximately right. Less memory, less memory traffic, faster decoding.

The “approximately” is where the engineering lives. Three things make Apertus harder to quantize than a run-of-the-mill Llama derivative:

1. It is not a Llama clone. Apertus uses the xIELU activation function and a non-gated MLP; there is no gate_proj module. Popular quantization tooling hardcodes LLaMA’s module layout, and it crashed on Apertus in two different code paths before we patched around it.

2. Some layers amplify quantization noise. xIELU squares its input, so 4-bit noise in the attention path gets amplified downstream. In our earlier experiments on the 8B model, quantizing attention to 4 bits produced NaN logits, a completely broken model. Attention needs gentler treatment.

3. The vocabulary is enormous. Apertus v1.5 has a 266,752-token vocabulary, because images are represented as discrete tokens in the same space as text (131k vision tokens + text + special tokens). The embedding and output layers tied to that vocabulary are quality-critical and disproportionately sensitive.

So a naive “quantize everything to 4 bits” produces either a crash or a lobotomized model. The answer is to treat different layers differently.

The three-tier recipe

The bulk of the weights is compressed to NVFP4, NVIDIA’s 4-bit floating-point format natively accelerated on the Blackwell GPU architecture; the noise-sensitive attention layers are kept at FP8; and the most fragile parts are left untouched at BF16. Vision and tool calling stay intact.

In other words, we quantized each group of layers as aggressively as it can tolerate, and no more:

The KV cache (the memory that grows with your context length) runs in FP8 as well, which doubles how much conversation fits on the card: the KV pool holds 207,760 tokens, a 192k-token context budget on a single card.

Two details we are particularly happy with:

• The complete two-pass conversion took approximately 3.4 minutes: 1.9 minutes for NVFP4 MLP quantization and 1.3 minutes for FP8 attention quantization. No calibration dataset, no cloud GPU rental, no shipping weights anywhere. The quantization streams the safetensors shards without ever loading the full model, which is what makes a 135 GiB model convertible on a machine with 64 GB of RAM.
• It is reproducible: the recipe is a short Python script using the open-source llm-compressor library.

Results

Measured on one NVIDIA RTX PRO 6000 Blackwell (96 GB), vLLM, identical prompts:

Context, precisely: with the FP8 KV cache at 90% GPU-memory utilization the model exposes a 207,760-token KV pool and runs a 192k max-model-len profile comfortably. Increasing utilization to 95% supports a measured 229k profile, but leaves essentially no operational headroom. The model’s full 262,144-token window does not fit on one 96 GiB card. FP8 context was not exhaustively profiled in the same run; approximately 14 GiB remains after loading weights, materially limiting context and multimodal headroom.

Performance and quality

The 48 GiB NVFP4 checkpoint decodes 55% faster than the 71 GiB FP8 variant (30.7 vs 19.8 TPS) while using 32% less weight storage.

On MMLU, FP8 scored 72.05% and NVFP4 scored 71.26%. That is a 0.79-point absolute difference, or 1.1% relative. The aggregate difference is within the combined statistical uncertainty of these runs. In practical terms, the 48 GiB checkpoint retains approximately 99% of FP8 benchmark quality while using 32% less weight storage and decoding 55% faster. The tradeoff is a small, measured 0.8-point MMLU difference.

Earlier RC-era regression tests also showed 93.7% token-level agreement and a median top-10 distribution divergence of 0.018 versus FP8. Those numbers were useful engineering validation of the quantization recipe, but the MMLU comparison above is the stronger publication result for the official-release checkpoint.

Multistream throughput

We compared NVFP4 and FP8 on a single RTX PRO 6000 Blackwell across generation throughput, prefill throughput, and max time-to-first-token (TTFT). Each concurrency level sends 10 requests with random 200-word prompts (~270 input tokens), generating 200 output tokens each. Generation speed is completion_tokens / wall_time, and prefill throughput is prompt_tokens / wall_time. The exact numbers are in the appendix table.

The two charts below visualize the same data: generation throughput first, then prefill throughput.

NVFP4 generation throughput exceeds FP8 by roughly 4% to 55% across the range (55% at concurrency 1, 37% at concurrency 8, 4% at concurrency 48). NVFP4 prefill throughput is consistently 11% to 15% lower, and NVFP4 max TTFT is 14% to 17% higher, reflecting the additional decoding speed at the cost of slightly slower prefill. At concurrency 56 and above the two formats essentially tie on generation throughput as the card’s memory bandwidth ceiling is reached. For the raw values behind every concurrency level, see the appendix table.

One card instead of two, with a lower rated GPU power envelope

The two-H200 reference deployment has a rated GPU board power of 2 x 600 W = 1,200 W (H200 NVL) to 2 x 700 W = 1,400 W (H200 SXM). The NVIDIA RTX PRO 6000 Blackwell is rated at 600 W total board power: one card instead of two, with roughly half the rated GPU power envelope. The comparison is based on NVIDIA-rated board power, not measured wall power. Actual system energy depends on utilization, CPUs, cooling, and power-supply efficiency.

And it is not a “runs, but slowly” compromise: because LLM decoding speed is limited by memory bandwidth, smaller weights decode faster. Our quantized model generates 30.7 tokens/s single-stream versus 19.8 tokens/s for the FP8 version of the same model on the same card.

At the same 600 W GPU power envelope, the throughput difference implies approximately 35% less energy per generated token. Actual energy use depends on measured board power and workload.

A community effort: from release to runnable

A model alone is not enough. The reason Apertus 1.5 runs smoothly on a single workstation GPU today is a distributed, volunteer-driven effort across open-source projects, organizations, and individual contributors.

The hardest piece was the serving stack. Standard vLLM did not understand Apertus’s Emu3.5 vision encoders and multimodal tokenizers. blancsw at Infomaniak spent months refactoring the multimodal pipeline so the encoders run GPU-natively inside the vLLM worker, with proper CPU and GPU separation, dynamic batch padding, and self-contained tokenizers. That GPU-native design is what makes single-card serving practical at all, because the vLLM memory profiler can finally see the encoders.

Building on that, Anunay-Yadav upstreamed the Apertus 1.5 support into the main vLLM repository as Apertus1p5ForConditionalGeneration, adding the correct architecture registration, the apertus tool parser, and the reasoning parser. Around that core, individual contributors added the pieces that make day-to-day use reliable: the ApertusToolParser from blancsw for native tool calling, the reasoning parser from AryanAhadinia, a double BOS-token fix from robmsmt, the original text-only vLLM integration with xIELU and QK-norm from EduardDurech and AllenHaoHuang, and a weight-loading fix for custom XIELU buffers from nathanrchn. Cyrilvallez at Swiss AI kept the HuggingFace Transformers integration in lockstep throughout.

Special thanks to Oleg Lavrovsky for fostering and coordinating the Apertus community ecosystem across GitHub, HuggingFace, and the developer community. His work connecting contributors, tracking upstream progress, and providing early feedback on tool parser development was key to making this collaboration happen.

Even the quantization recipe is community tooling. Neural Magic’s llm-compressor is what turns the 135 GiB BF16 checkpoint into the 48 GiB NVFP4 model this post is about, and we used the llmapibenchmark tool to validate throughput and latency at realistic concurrency.

Several of these fixes landed directly in the official v1.5 release, so tool calling and chat-template handling work out of the box for everyone who downloads the model. That is open source working as intended.

Thanks to blancsw and Infomaniak for the GPU-native branch, to Anunay-Yadav for upstreaming it, and to everyone else who debugged, reviewed, and benchmarked along the way.

The Bigger Picture: Sovereign AI Within Reach

A year ago, “run a 70B multimodal model” meant a multi-GPU datacenter node. Today the best open Swiss model runs on one card you can put in a server under your desk: sovereign, air-gap capable, with vision, tool calling and a 192k context profile (229k measured maximum), at roughly half the rated GPU power of the reference deployment, approximately 35% less energy per generated token than the FP8 alternative, and 99% of FP8 MMLU quality in a 48 GiB checkpoint.

The weights recipe, the numbers, and the fixes are public. If you download the official Apertus v1.5 release, tool calling already works out of the box, partly because of the round trip described in this post. That is the quiet superpower of open models: everyone’s deployment problems become everyone’s fixes.

Thanks to Infomaniak / @blancsw for the GPU-native Apertus vLLM work, and to the Apertus team for rapid upstream adoption during the release cycle.

Appendix

Multistream throughput data

Measured on one NVIDIA RTX PRO 6000 Blackwell (96 GB), vLLM, identical prompts. Each concurrency level sends 10 requests with random 200-word prompts (~270 input tokens), generating 200 output tokens each. Generation speed is completion_tokens / wall_time, and prefill throughput is prompt_tokens / wall_time.