vLLM

Downloading a model’s weights is one thing, serving them quickly to lots of users is another. vLLM is free software that sits between the weights and your application: it loads the model, batches incoming requests together, manages GPU memory carefully, and exposes an interface that imitates OpenAI’s API, so existing tools work against it unchanged.

If you read a model card that says “we recommend using this model with vLLM”, this is what it means. The main alternatives are SGLang, which competes on similar ground, and llama.cpp, which targets small machines and laptops rather than servers.