llama.cpp build 11899: giving multimodal clients an interface to ask what a model supports

llama.cpp’s b11899 is the October 26 nightly. Nightlies usually carry one or two changes, and this one’s body is short:

mtmd: add mtmd_helper_get_capabilities (#31183) — adds mtmd_helper_get_capabilities, covers audio_stream, keeps the old API for compatibility, adds audio_gen_spk_ref and text_gen.

Start with the build number

b11899 means the 11,899th build. llama.cpp publishes with incrementing build numbers, essentially daily, each with binaries for a dozen or more platforms. That is more than “frequent releases” — it changes how you use it:

• you do not read changelogs looking for features, you pull the latest when you need it;
• when something breaks, the first question is “which build?”;
• upstream fixes propagate in hours.

For an inference library that is reasonable: its upstream inputs — model formats, GPU backends, quantization algorithms — are themselves moving fast.

What mtmd is, and what capability probing solves

mtmd is llama.cpp’s multimodal layer: it turns images, audio and text into a sequence the model can consume. The hard part is not the assembling — it is that different models want completely different input shapes:

Model kind	Input	Capability needed
Text only	tokens	text_gen
Vision	image patches	image encoder plus projector
Speech input	audio frames	audio_stream
Speech output	speaker reference plus text	audio_gen_spk_ref

Clients used to guess from the model name or file metadata: can this model take audio? Does it emit speech with a speaker timbre? Guessing wrong does not raise an error — it silently produces garbage. Feed audio where an image is expected and the model still answers with text.

mtmd_helper_get_capabilities turns the guess into a query. The three flags named in the release note are the whole story:

• audio_stream: can it accept streamed audio;
• audio_gen_spk_ref: can it synthesize speech from a reference voice (voice cloning style);
• text_gen: can it generate text.

“Keep the old API” deserves its own paragraph

The PR explicitly keeps the old API for compatibility. In a daily-build project that is not trivial — upstream may break compatibility whenever it likes, on the grounds that anyone on the newest build is on their own. Keeping the old entry point means the maintainers treat mtmd as a public layer with external consumers, not an internal module.

That is exactly how I judge whether a C or C++ library is worth depending on: does it distinguish internal implementation from an external contract? llama.cpp’s core API has churned (argument structs changed several times), but compatibility at the llama.h level has held up; mtmd now enters that stage.

Supply chain: attestations

The release page carries something easy to skip: an Attestations link pointing at GitHub build provenance — which workflow, which commit, which environment produced these binaries.

For a library you download as binaries, provenance is more useful than a signature: a signature proves “official”, an attestation proves “built from this commit by this workflow”. If you plan to run prebuilt binaries in production, check this first.

How I would use it

I do not chase nightlies. My rules:

1. Track versions, not builds: upgrade when a build fixes something I actually hit, not every day.
2. Capability probing beats model names: if your code handles multimodal input, call mtmd_helper_get_capabilities first instead of judging by filename.
3. Watch attestations: verify build provenance before downloading, which matters most with dozens of artifacts under one build number — any single file could be swapped and nobody would notice.

llama.cpp is where ffmpeg was a decade ago: every product doing local inference depends on it, and nobody wants to read its source. What matters in such a project is not new features but whether the external contract holds — and b11899 is a small step in that direction.