AI models

Parameters refer to the number of learnable weights in the model (measured in billions, e.g., 70B). Size refers to the storage space required for the model files (measured in GB). Quantized models have fewer bits per parameter, resulting in smaller file sizes while maintaining most of the model's capabilities.

Allow VRAM for the model weights, KV cache and inference runtime. Requirements depend on quantization, context length and concurrent requests, so the model file size alone is not enough to size a server. Quantizing weights or KV cache can reduce memory use. Validate memory consumption and output quality with your intended workload.

Experimental models are newer quantizations or configurations that are still being validated. They may offer better performance or efficiency but haven't been thoroughly tested in production environments. Stable models have been verified for reliable operation.

Start with the models, quantization and number of concurrent users you need. Use the server selector to compare compatible model families, then check the specific variant. Allow additional VRAM for context and KV cache, not just the model files.

Yes, if you have sufficient VRAM. The total size of loaded models must fit within your server's available memory. Larger tiers allow running several models concurrently for different use cases.

Modalities indicate what types of data a model can process (input) and generate (output). Text models handle written content, image models can analyze or generate visuals, code models are optimized for programming tasks, and multimodal models combine multiple capabilities.