Clarification on Runtime Structure (vLLM/llama.cpp, CUDA Images)

[HannesDampft]

Hello everyone,

we are currently using Ramalama to host LLM workloads in Kubernetes, where each instance is isolated per pod. At the moment, we only perform inference with a single GPU and exclusively use the llama.cpp runtime.

As we now plan to transition to multi‑GPU deployments, our goal is to adopt vLLM as the backend, given its superior performance with tensor parallelism. However, I’ve encountered some uncertainty regarding how Ramalama structures its container images.

Specifically:

1. There appear to be several related images, but the cuda‑vllm image is not published on quay.io .
2. The ramalama serve command is currently failing in our test setup. (supposedly fixed in Inference engine spec #1959)
3. It’s unclear to me how Ramalama is designed to provide and switch between different inference runtimes inside the container.

Could someone clarify how the runtime selection and container layering are intended to work? I’d be happy to assist in improving the documentation around this area and can prepare a PR for review if that would be helpful.

Thanks in advance for any insights!
Hans

[federicofortini]

Hello! I'm not sure I understood the question correctly, but i'll try to anwer anyway, mybe I'll ve lucky, In the git repo i found this: https://github.com/containers/ramalama/blob/main/container-images/cuda-vllm/Containerfile
could this help?