Srihari Sriraman shared in

Public

Built and tested llama.cpp with Gemma modelPublic

Agent

Model

GPT-5.4

Started

Apr 2, 7:22 PM

Messages103

Downloads0

Summary

Closed, Completed

Build
Built llama.cpp from source.
Build
Setup
Started llama-server with Gemma model.
Setup
Verify
Tested model output and found garbage tokens.
Verify
Cleanup
Deleted llama.cpp source and model cache.
Cleanup

/var/folders/49/l4171l3n219_xwq30fmyzr8w0000gn/T/pi-clipboard-0c15628d-b04b-4323-a156-17b60d3e00dc.png wanna try this out. figure out how to get the latest llama cpp going, ideally we compile from source.

Yep. The screenshot flow is plausible, but I would tighten it a bit so pi can target a stable model ID.

For macOS, this is the clean path to get the latest llama.cpp from source and expose it to pi over the OpenAI-compatible server.

1. Build llama.cpp from source

Requirements:

Then:

Binary should end up at:

For Apple Silicon, GGML_METAL=ON is the important part.

2. Start the server with a stable alias

Your screenshot uses:

I would run it like this instead:

Why --alias matters:

pi sends a model string to the OpenAI-compatible endpoint
using a short stable alias avoids guessing what llama.cpp exposes internally
your models.json can use that exact alias

If you want the server reachable from other machines on your LAN, use --host 0.0.0.0 instead.

3. Verify the server

Check that the OpenAI-compatible API is up:

You should see the alias in the response.

Then a quick chat test:

bash

curl http://127.0.0.1:8080/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{    "model": "gemma-4-26b-a4b-it-q4km",    "messages": [      { "role": "user", "content": "say hello" }    ],    "stream": false  }'

If that works, pi can use it.

4. Configure pi

Create ~/.pi/agent/models.json:

json

{  "providers": {    "llama-cpp": {      "baseUrl": "http://127.0.0.1:8080/v1",      "api": "openai-completions",      "apiKey": "dummy",      "compat": {        "supportsDeveloperRole": false      },      "models": [        {          "id": "gemma-4-26b-a4b-it-q4km",          "name": "Gemma 4 26B A4B IT Q4_K_M (llama.cpp local)",          "reasoning": false,          "input": ["text"]        }      ]    }  }}

Notes:

apiKey is required by pi config shape, but local llama-server does not care
supportsDeveloperRole: false is a good default for OpenAI-compatible local servers
if this specific model supports images through llama.cpp later, you can change input to ["text", "image"]

5. Use it in pi

Then launch pi and select it with /model, or directly:

6. If pi or llama.cpp is picky

If requests fail, the usual compatibility knobs for local OpenAI-compatible servers are:

I would start with only supportsDeveloperRole: false, then add others only if needed.

7. Useful operational notes

Update to latest llama.cpp

Pick another quant

You can swap the model source without changing pi config, as long as you keep the same alias:

Though in practice I would rename the alias to match the quant.

Keep it running

For regular use, run it in tmux:

Recommended final setup

If you want the shortest path, use exactly this:

Build

Run

pi config

json

{  "providers": {    "llama-cpp": {      "baseUrl": "http://127.0.0.1:8080/v1",      "api": "openai-completions",      "apiKey": "dummy",      "compat": {        "supportsDeveloperRole": false      },      "models": [        { "id": "gemma-4-26b-a4b-it-q4km" }      ]    }  }}

If you want, I can next turn this into:

a copy-paste shell script that installs and builds it, or
the exact models.json for a few candidate local models, not just Gemma 4.

ok, build llama first, and test it works with the gemma model

Building llama.cpp, then I’ll start llama-server with the Gemma GGUF and verify the OpenAI-compatible endpoint.

llama.cpp built successfully at ~/src/llama.cpp/build/bin/llama-server. Now I’m launching it against the Gemma GGUF and probing /v1/models.

how we going?

Build is done. The Gemma test is not up yet.

Current state:

llama.cpp built successfully
llama-server binary exists at ~/src/llama.cpp/build/bin/llama-server
launching with -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M is not reaching port 8080
log only shows Metal init plus:
- common_download_file_single_online: HEAD failed, status: 404
- no remote preset found, skipping

That preset line is not fatal by itself. The likely issue is the Hugging Face model resolution or download step is hanging before the server binds.

I can continue by checking:

whether the repo name is still correct
whether that exact quant exists
whether llama-server wants --hf-file for this repo layout
whether we should pre-download the GGUF first and run with -m <local-file>

That is the route I’d take next.

Build succeeded.

Blocked part:

llama-server is running as PID 31038
it is not listening on 127.0.0.1:8080 yet
log is stuck after Metal init with:
- common_download_file_single_online: HEAD failed, status: 404
- no remote preset found, skipping

So the issue is not the compile. It is model fetch/startup for ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M.

Next step is to stop relying on auto-resolution and verify the exact HF artifact, then either:

launch with -hf ... --hf-file ..., or
pre-download the GGUF and run -m /path/to/model.gguf

I can do that next and get it to a working /v1/models response.

i thin 8080 is assigned to a different process? check

Checking what owns port 8080, then I’ll either kill the stale llama-server or move to a free port.

Port 8080 is free.

Results:

no listener on TCP:8080
no open FDs on 8080
the llama-server process is still alive, but it never bound the port

So the problem is upstream of socket bind, likely model resolution/download/startup, not port contention.

is llama-server process still running and downloading?

I’m checking whether the llama-server process is still alive and whether it shows active download or file write activity.

Yes.

Current state:

process 31038 is still running
it has an active HTTPS connection open to 99.84.91.56:443
it has a live download temp file open:

/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress

That file is already about 5.89 GB, so it is definitely in the download path.

There is also a real issue in the log:

unable to rename file: ...4107c1c3....downloadInProgress to ...4107c1c3...

So it looks like:

main GGUF download is active
another downloaded artifact hit a rename/cache problem
server has not bound the port yet because model setup is still in progress or partially stuck on that cache step

If you want, I can monitor until the GGUF finishes, then retry the server cleanly. Or I can stop it now, inspect the Hugging Face cache layout, and relaunch in a cleaner way using a pre-downloaded local model path.

update?

Checking current download size, process state, and whether the server bound yet.

Current status:

llama-server is still running
it still has not bound port 8080
the Gemma GGUF is actively downloading
download size is now 11.05 GB

Relevant files:

active download:
- ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997....downloadInProgress
another blob is already present:
- ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3...
- size 2.08 GB

Log has not advanced beyond the earlier cache rename warning, but the open file growth confirms the download is still moving.

So:

build: done
port issue: not the problem
model fetch: still in progress

If you want, I can keep polling until the download completes, then verify /v1/models and run a test prompt.

update

Polling current state of the Gemma download and server bind.

Update:

still downloading
still not listening on 8080
active GGUF temp file is now 25.77 GB

So it is making substantial progress. No new error beyond the earlier rename warning.

Status:

process alive
network active
file growing
server not ready yet

This model/quant is much larger than expected, so startup is gated on that full artifact download.

now?

Checking whether the download finished and whether llama-server finally bound the API port.

Now we have the failure point.

Status:

download completed
llama-server exited
port 8080 is still free
final error:

error: failed to download model from Hugging Face

The cache now contains:

~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c699...
size 33.35 GB
plus another blob at 2.08 GB

So the model artifact is on disk, but llama-server tripped over its cache rename/finalization logic and aborted.

Best next move:

stop using -hf ...
resolve the actual local GGUF path from the HF cache
launch llama-server -m <local-gguf> directly

That should bypass the broken download-finalization path entirely. If you want, I’ll do that next and get the server actually responding.

try again

Retrying now using the already downloaded local cache file instead of -hf, then I’ll probe /v1/models and a test completion.

Found the cached GGUFs. One is a 31 GB main model, the other is a 1.9 GB auxiliary GGUF. I’m identifying which one is the actual text model, then I’ll start llama-server with the local file path.

Server is up on 127.0.0.1:8080. I’m running one actual chat completion against the Gemma alias now.

The server works, but the first OpenAI chat response is garbage tokens (<unused25>...). I’m doing one more probe with simpler generation settings to see if this is just a chat-template issue.

Retried successfully with the local cached GGUFs instead of -hf.

Current state:

llama-server is running
listening on http://127.0.0.1:8080
/v1/models works

Launch command that worked:

bash

~/src/llama.cpp/build/bin/llama-server \  -m ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 \  --mmproj ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

Verified:

model load completed
server bound port 8080
/v1/models returns gemma-4-26b-a4b-it-q4km

But generation is not usable yet:

/v1/chat/completions returns repeated <unused25>
/completion and /v1/completions do the same

So the server stack is working, but this Gemma 4 GGUF combo is not producing sane text in the current llama.cpp run. That points to one of:

a Gemma 4 support issue/regression in current llama.cpp
this specific GGUF needing different runtime flags
a model/template/tokenizer mismatch for this release

Files/logs:

server log: /tmp/llama-pi/gemma-local.log
pid file: /tmp/llama-pi/gemma-local.pid

If you want, next I can:

try alternate flags for Gemma 4 in llama.cpp
try a different Gemma 4 quant/repo
wire pi to the working endpoint anyway, then swap to a model that actually generates sane output

what does the model output?

It outputs garbage placeholder tokens.

Examples:

/v1/chat/completions

json

{  "message": {    "role": "assistant",    "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"  }}

/completion

json

{  "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"}

So:

server loads
API responds
actual text generation is broken for this model setup

Not a port/build issue anymore. It is a model/runtime correctness issue.

ok, delete llama cpp and the model cache shit

Removing:

~/src/llama.cpp
Hugging Face cache for ggml-org/gemma-4-26b-a4b-it-GGUF

I’ll stop any running llama-server first.

Removed:

~/src/llama.cpp
~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF
/tmp/llama-pi

Any launched llama-server processes from this session were stopped first.