Built and tested llama.cpp with Gemma model

Summary

Closed, Completed

Build
Built llama.cpp from source.
Build
Setup
Started llama-server with Gemma model.
Setup
Verify
Tested model output and found garbage tokens.
Verify
Cleanup
Deleted llama.cpp source and model cache.
Cleanup

/var/folders/49/l4171l3n219_xwq30fmyzr8w0000gn/T/pi-clipboard-0c15628d-b04b-4323-a156-17b60d3e00dc.png wanna try this out. figure out how to get the latest llama cpp going, ideally we compile from source.

Inspecting image for instructions

I need to check out the image the user mentioned, as they're trying to figure out how to get the latest LLaMA CPP compiled from source. I wonder if this image could have relevant details—I should inspect it carefully, but I don't want to modify anything unless the user asks me to. I’ll use the read tool to see what's in the image and discuss any steps if necessary. Let's take a look!

Readpi-clipboard-0c15628d-b04b-4323-a156-17b60d3e00dc.png

Exploring documentation needs

I’m thinking I might need to read through the docs, especially models.md and the README file, since they seem important for the topic related to pi. I wonder if just reading models.json usage will be sufficient, or if I should dive into the custom-provider details too. To be on the safe side, I'll check both models.md and the coding-agent README fully. It seems like a thorough approach!

Readmodels.md

ReadREADME.md

Considering Llama CPP Setup

I think we have enough documentation to consider compiling from source with Metal/CUDA, especially since the user is likely on a Mac. I should ask if they want help with installation or building. There wasn’t a specific request for implementation, but it seems like they want assistance setting it up, so I could provide the steps and relevant commands. I might also want to include an example, including the screenshot command and mention the model ID for the llama-server.

Reviewing Llama CPP Compilation

I’m considering that in the OpenAI endpoint, the model ID might relate to a repository. It could be helpful to verify this with the llama.cpp documentation, though I might not have internet access. I could check what’s already installed, potentially using commands like git ls-remote. For macOS, I want to outline the Metal build commands. I think I should include commands like git clone, cmake settings, and the generated binary path. I’ll also mention the correct model ID in the models.json.

Setting Up Model Aliases

I just realized that the OpenAI requests require a model field, and the llama-server can use a --alias argument to set the model name, which is really helpful! I should recommend running the server with --alias gemma-4-26b-a4b-it-q4km, and then that ID should go into the models.json. I’m also thinking it might be good to mention using --jinja for template purposes as well. This feels like a solid plan to provide for the user!

Preparing model setup instructions

I need to mention a few key steps for setting up the model. First, clone and build the project, then run the server with the specified command. Testing can be done through curl commands to check models and completions. I should include installation of dependencies like CMake and Xcode CLI tools. If users want GPU support on NVIDIA or AMD, I’ll explain the necessary options, but generally, they may be on Mac. Maybe I should also clarify if anyone wants help with the compilation process.

Yep. The screenshot flow is plausible, but I would tighten it a bit so pi can target a stable model ID.

For macOS, this is the clean path to get the latest llama.cpp from source and expose it to pi over the OpenAI-compatible server.

1. Build llama.cpp from source

Requirements:

Then:

bash

cd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j

cd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j

Binary should end up at:

For Apple Silicon, GGML_METAL=ON is the important part.

2. Start the server with a stable alias

Your screenshot uses:

I would run it like this instead:

bash

~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

Why --alias matters:

pi sends a model string to the OpenAI-compatible endpoint
using a short stable alias avoids guessing what llama.cpp exposes internally
your models.json can use that exact alias

If you want the server reachable from other machines on your LAN, use --host 0.0.0.0 instead.

3. Verify the server

Check that the OpenAI-compatible API is up:

You should see the alias in the response.

Then a quick chat test:

bash

curl http://127.0.0.1:8080/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{    "model": "gemma-4-26b-a4b-it-q4km",    "messages": [      { "role": "user", "content": "say hello" }    ],    "stream": false  }'

curl http://127.0.0.1:8080/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{    "model": "gemma-4-26b-a4b-it-q4km",    "messages": [      { "role": "user", "content": "say hello" }    ],    "stream": false  }'

If that works, pi can use it.

4. Configure pi

Create ~/.pi/agent/models.json:

json

{  "providers": {    "llama-cpp": {      "baseUrl": "http://127.0.0.1:8080/v1",      "api": "openai-completions",      "apiKey": "dummy",      "compat": {        "supportsDeveloperRole": false      },      "models": [        {          "id": "gemma-4-26b-a4b-it-q4km",          "name": "Gemma 4 26B A4B IT Q4_K_M (llama.cpp local)",          "reasoning": false,          "input": ["text"]        }      ]    }  }}

{  "providers": {    "llama-cpp": {      "baseUrl": "http://127.0.0.1:8080/v1",      "api": "openai-completions",      "apiKey": "dummy",      "compat": {        "supportsDeveloperRole": false      },      "models": [        {          "id": "gemma-4-26b-a4b-it-q4km",          "name": "Gemma 4 26B A4B IT Q4_K_M (llama.cpp local)",          "reasoning": false,          "input": ["text"]        }      ]    }  }}

Notes:

apiKey is required by pi config shape, but local llama-server does not care
supportsDeveloperRole: false is a good default for OpenAI-compatible local servers
if this specific model supports images through llama.cpp later, you can change input to ["text", "image"]

5. Use it in pi

Then launch pi and select it with /model, or directly:

6. If pi or llama.cpp is picky

If requests fail, the usual compatibility knobs for local OpenAI-compatible servers are:

json

"compat": {  "supportsDeveloperRole": false,  "supportsReasoningEffort": false,  "supportsUsageInStreaming": false,  "maxTokensField": "max_tokens"}

"compat": {  "supportsDeveloperRole": false,  "supportsReasoningEffort": false,  "supportsUsageInStreaming": false,  "maxTokensField": "max_tokens"}

I would start with only supportsDeveloperRole: false, then add others only if needed.

7. Useful operational notes

Update to latest llama.cpp

Pick another quant

You can swap the model source without changing pi config, as long as you keep the same alias:

bash

~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q5_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q5_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

Though in practice I would rename the alias to match the quant.

Keep it running

For regular use, run it in tmux:

bash

tmux new -s llama~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

tmux new -s llama~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

Recommended final setup

If you want the shortest path, use exactly this:

Build

bash

xcode-select --installbrew install cmakecd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j

xcode-select --installbrew install cmakecd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j

Run

bash

~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

~/src/llama.cpp/build/bin/llama-server \  -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

pi config

json

{  "providers": {    "llama-cpp": {      "baseUrl": "http://127.0.0.1:8080/v1",      "api": "openai-completions",      "apiKey": "dummy",      "compat": {        "supportsDeveloperRole": false      },      "models": [        { "id": "gemma-4-26b-a4b-it-q4km" }      ]    }  }}

{  "providers": {    "llama-cpp": {      "baseUrl": "http://127.0.0.1:8080/v1",      "api": "openai-completions",      "apiKey": "dummy",      "compat": {        "supportsDeveloperRole": false      },      "models": [        { "id": "gemma-4-26b-a4b-it-q4km" }      ]    }  }}

If you want, I can next turn this into:

a copy-paste shell script that installs and builds it, or
the exact models.json for a few candidate local models, not just Gemma 4.

ok, build llama first, and test it works with the gemma model

Planning implementation steps

I need to implement a concrete task, and I should check if there’s a README for guidance. I might consider using the bash command for tasks like ls, and I should check if the /src directory exists. Compiling the source could take some time and require internet, so I need to be cautious about permissions and possibly avoid using sudo. I might also need to see if cmake is already installed. Let’s go through these steps with clear commentary.

Building llama.cpp, then I’ll start llama-server with the Gemma GGUF and verify the OpenAI-compatible endpoint.

llama.cpp built successfully at ~/src/llama.cpp/build/bin/llama-server. Now I’m launching it against the Gemma GGUF and probing /v1/models.

Adjusting command usage

I realized that the "rg" pattern requires either the -e flag or quotes with --. I see that maybe there's a way to simplify using grep, but I must use "rg" instead of grep. I’ll need to work with bash and consider redirecting output to a file. I can also pipe to "rg" with a command like 'alias|host' to get the results I need. It feels like I'm getting closer to figuring this out!

Planning a server run

I’m considering running a server, potentially using nohup to keep it in the background, especially since it might involve a huge download. It’s a good idea to set a timeout of about 20 minutes and poll the progress while it runs. I might need to check the cache location too. Using default context size seems best, but I’ll set an alias for convenience. There’s a chance it might fail due to memory issues, so I’m staying cautious!

>mkdir -p /tmp/llama-pi && cd ~/src/llama.cpp && rm -f /tmp/llama-pi/gemma-server.log /tmp/llama-pi/gemma-server.pid && nohup ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080 > /tmp/llama-pi/gemma-server.log 2>&1 & echo $! > /tmp/llama-pi/gemma-server.pid && echo "pid=$(cat /tmp/llama-pi/gemma-server.pid)" && sleep 5 && lsof -nP -iTCP:8080 -sTCP:LISTEN || true && echo '--- LOG TAIL ---' && tail -n 80 /tmp/llama-pi/gemma-server.log

Considering shell commands

I'm thinking about a possible issue with grouping commands like mkdir -p and rm. It seems like the redirection happens before mkdir gets executed in a subshell, which could lead to a missing path. I wonder if separating them with a semicolon would be safer, and maybe referencing $HOME might help too. It's essential to clarify and ensure that the commands work as intended! I want to get this right!

>mkdir -p /tmp/llama-pi; cd ~/src/llama.cpp; nohup ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080 >/tmp/llama-pi/gemma-server.log 2>&1 & echo $! >/tmp/llama-pi/gemma-server.pid; echo "pid=$(cat /tmp/llama-pi/gemma-server.pid)"; sleep 5; lsof -nP -iTCP:8080 -sTCP:LISTEN || true; echo '--- LOG TAIL ---'; tail -n 80 /tmp/llama-pi/gemma-server.log

how we going?

Inspecting server status

I need to check the server's status because the build seems okay, but it’s either stuck downloading or there may have been a failure with the preset. I should investigate the running process to see if it’s actually listening. There might not be any changes right now, so it could be helpful to use Bash to perform a deeper inspection and figure out what’s going on. Let's see what I can find!

Build is done. The Gemma test is not up yet.

Current state:

llama.cpp built successfully
llama-server binary exists at ~/src/llama.cpp/build/bin/llama-server
launching with -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M is not reaching port 8080
log only shows Metal init plus:
- common_download_file_single_online: HEAD failed, status: 404
- no remote preset found, skipping

That preset line is not fatal by itself. The likely issue is the Hugging Face model resolution or download step is hanging before the server binds.

I can continue by checking:

whether the repo name is still correct
whether that exact quant exists
whether llama-server wants --hf-file for this repo layout
whether we should pre-download the GGUF first and run with -m <local-file>

That is the route I’d take next.

Considering user request

I’m thinking about how to respond to the user. They asked for a status update, which means I should keep it concise and focused. I could consider using web tools, but it seems like they want the status rather than detailed information. It might be worth noting that the process is still running but likely blocked. I want to ensure I'm addressing their request effectively without overcomplicating things.

Build succeeded.

Blocked part:

llama-server is running as PID 31038
it is not listening on 127.0.0.1:8080 yet
log is stuck after Metal init with:
- common_download_file_single_online: HEAD failed, status: 404
- no remote preset found, skipping

So the issue is not the compile. It is model fetch/startup for ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M.

Next step is to stop relying on auto-resolution and verify the exact HF artifact, then either:

launch with -hf ... --hf-file ..., or
pre-download the GGUF and run -m /path/to/model.gguf

I can do that next and get it to a working /v1/models response.

i thin 8080 is assigned to a different process? check

Inspecting port usage

I need to inspect what's happening on port 8080. My best plan is to use the lsof command, which can help me see which processes are using that port. It will provide a list of all the open files and their associated processes, giving me clarity on who’s listening on that port or if there’s any related activity. It's a handy tool for troubleshooting and keeping things running smoothly!

Checking what owns port 8080, then I’ll either kill the stale llama-server or move to a free port.

Port 8080 is free.

Results:

no listener on TCP:8080
no open FDs on 8080
the llama-server process is still alive, but it never bound the port

So the problem is upstream of socket bind, likely model resolution/download/startup, not port contention.

is llama-server process still running and downloading?

I’m checking whether the llama-server process is still alive and whether it shows active download or file write activity.

>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
if [ -z "$PID" ]; then echo 'no pid file'; exit 0; fi

echo '--- process ---'
ps -p "$PID" -o pid,ppid,state,etime,%cpu,%mem,command || true

echo '--- open files mentioning cache/huggingface/gguf ---'
lsof -p "$PID" 2>/dev/null | rg -i 'hugging|cache|gguf|tmp|download' || true

echo '--- network sockets ---'
lsof -nP -p "$PID" -i || true

echo '--- child processes ---'
pgrep -P "$PID" -fl . || true

echo '--- likely cache dirs ---'
for d in "$HOME/.cache/llama.cpp" "$HOME/.cache/huggingface" "$HOME/Library/Caches/llama.cpp" "$HOME/Library/Caches/ggml"; do
  if [ -e "$d" ]; then
    echo "# $d"
    find "$d" -type f -mmin -10 2>/dev/null | head -n 20
  fi
done

echo '--- recent log ---'
tail -n 80 /tmp/llama-pi/gemma-server.log 2>/dev/null || true

Revising response clarity

I need to provide a concise answer about running and downloading, ensuring I mention the evidence. It’s also a good idea to rename any errors to avoid confusion.

Let’s see if I can streamline my message! I want it to be clear and informative while also being brief. I’ll focus on the essential points and make sure there’s no ambiguity in what I’m saying. Getting this right is important!

Yes.

Current state:

process 31038 is still running
it has an active HTTPS connection open to 99.84.91.56:443
it has a live download temp file open:

/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress

That file is already about 5.89 GB, so it is definitely in the download path.

There is also a real issue in the log:

unable to rename file: ...4107c1c3....downloadInProgress to ...4107c1c3...

So it looks like:

main GGUF download is active
another downloaded artifact hit a rename/cache problem
server has not bound the port yet because model setup is still in progress or partially stuck on that cache step

If you want, I can monitor until the GGUF finishes, then retry the server cleanly. Or I can stop it now, inspect the Hugging Face cache layout, and relaunch in a cleaner way using a pre-downloaded local model path.

update?

Checking current download size, process state, and whether the server bound yet.

>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
echo '--- process ---'
ps -p "$PID" -o pid,state,etime,%cpu,%mem,command || true

echo '--- 8080 ---'
lsof -nP -iTCP:8080 -sTCP:LISTEN || true

echo '--- active download files ---'
lsof -p "$PID" 2>/dev/null | rg 'downloadInProgress|models--ggml-org--gemma-4-26b-a4b-it-GGUF' || true

echo '--- cache file sizes ---'
find "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs" -maxdepth 1 -type f 2>/dev/null | xargs -I{} stat -f '%z %N' '{}' 2>/dev/null | sort -nr | head -n 10 || true

echo '--- log tail ---'
tail -n 60 /tmp/llama-pi/gemma-server.log 2>/dev/null || true

Current status:

llama-server is still running
it still has not bound port 8080
the Gemma GGUF is actively downloading
download size is now 11.05 GB

Relevant files:

active download:
- ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997....downloadInProgress
another blob is already present:
- ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3...
- size 2.08 GB

Log has not advanced beyond the earlier cache rename warning, but the open file growth confirms the download is still moving.

So:

build: done
port issue: not the problem
model fetch: still in progress

If you want, I can keep polling until the download completes, then verify /v1/models and run a test prompt.

update

Polling current state of the Gemma download and server bind.

>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
echo '--- process ---'
ps -p "$PID" -o pid,state,etime,%cpu,%mem,command || true

echo '--- 8080 ---'
lsof -nP -iTCP:8080 -sTCP:LISTEN || true

echo '--- download file ---'
lsof -p "$PID" 2>/dev/null | rg 'downloadInProgress|models--ggml-org--gemma-4-26b-a4b-it-GGUF' || true

echo '--- latest blob sizes ---'
find "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs" -maxdepth 1 -type f 2>/dev/null | xargs -I{} stat -f '%Sm %z %N' -t '%Y-%m-%d %H:%M:%S' '{}' 2>/dev/null | sort -r | head -n 12 || true

echo '--- log tail ---'
tail -n 80 /tmp/llama-pi/gemma-server.log 2>/dev/null || true

Update:

still downloading
still not listening on 8080
active GGUF temp file is now 25.77 GB

So it is making substantial progress. No new error beyond the earlier rename warning.

Status:

process alive
network active
file growing
server not ready yet

This model/quant is much larger than expected, so startup is gated on that full artifact download.

now?

Checking whether the download finished and whether llama-server finally bound the API port.

>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
echo '--- process ---'
ps -p "$PID" -o pid,state,etime,%cpu,%mem,command || true

echo '--- 8080 ---'
lsof -nP -iTCP:8080 -sTCP:LISTEN || true

echo '--- curl /v1/models ---'
curl -sS http://127.0.0.1:8080/v1/models || true

echo '
--- cache status ---'
find "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs" -maxdepth 1 -type f 2>/dev/null | xargs -I{} stat -f '%Sm %z %N' -t '%Y-%m-%d %H:%M:%S' '{}' 2>/dev/null | sort -r | head -n 12 || true

echo '--- open download files ---'
lsof -p "$PID" 2>/dev/null | rg 'downloadInProgress|models--ggml-org--gemma-4-26b-a4b-it-GGUF' || true

echo '--- log tail ---'
tail -n 120 /tmp/llama-pi/gemma-server.log 2>/dev/null || true

Now we have the failure point.

Status:

download completed
llama-server exited
port 8080 is still free
final error:

error: failed to download model from Hugging Face

The cache now contains:

~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c699...
size 33.35 GB
plus another blob at 2.08 GB

So the model artifact is on disk, but llama-server tripped over its cache rename/finalization logic and aborted.

Best next move:

stop using -hf ...
resolve the actual local GGUF path from the HF cache
launch llama-server -m <local-gguf> directly

That should bypass the broken download-finalization path entirely. If you want, I’ll do that next and get the server actually responding.

try again

Retrying now using the already downloaded local cache file instead of -hf, then I’ll probe /v1/models and a test completion.

>set -e
cd ~/src/llama.cpp

CACHE_DIR="$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF"

echo '--- cache layout ---'
find "$CACHE_DIR" -maxdepth 3 \( -type f -o -type l \) | sort | head -n 200

echo '--- refs/main ---'
if [ -f "$CACHE_DIR/refs/main" ]; then cat "$CACHE_DIR/refs/main"; fi

echo '--- snapshots ---'
find "$CACHE_DIR/snapshots" -maxdepth 2 \( -type f -o -type l \) 2>/dev/null | sort || true

Inspecting blob types

I need to check the blob types, maybe looking for strings. I’m wondering if one could be for “gguf” and the other for a tokenizer. To get a clearer understanding, I’ll use a file command. It's a straightforward approach, but I want to ensure I correctly identify each type. This way, I can manage the data appropriately, ensuring I’m not missing anything important. Let's get started on that!

Considering project specifications

I’m thinking I might need one mmproj. For the launch, it seems like the 31G main model could be enough—maybe even multimodal? I wonder if gemma-4 needs a projector or if it could just be text-only. I’m curious about "a4b-it" in relation to text. Should I consider splitting into two gguf? I should also inspect the metadata—maybe with llama-gguf, as it could provide the outputs with architecture names.

Found the cached GGUFs. One is a 31 GB main model, the other is a 1.9 GB auxiliary GGUF. I’m identifying which one is the actual text model, then I’ll start llama-server with the local file path.

>set -e
cd ~/src/llama.cpp
MODEL="$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5"
MMPROJ="$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531"
rm -f /tmp/llama-pi/gemma-local.log /tmp/llama-pi/gemma-local.pid
nohup ./build/bin/llama-server -m "$MODEL" --mmproj "$MMPROJ" --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080 >/tmp/llama-pi/gemma-local.log 2>&1 &
echo $! >/tmp/llama-pi/gemma-local.pid
sleep 8
printf 'pid='; cat /tmp/llama-pi/gemma-local.pid
echo '--- log tail ---'
tail -n 120 /tmp/llama-pi/gemma-local.log

>for i in $(seq 1 18); do
  echo "== poll $i =="
  ps -p $(cat /tmp/llama-pi/gemma-local.pid) -o pid,state,etime,%cpu,%mem,command || true
  lsof -nP -iTCP:8080 -sTCP:LISTEN || true
  if curl -sf http://127.0.0.1:8080/v1/models >/tmp/llama-pi/models.json 2>/dev/null; then
    echo '--- models ---'
    cat /tmp/llama-pi/models.json
    break
  fi
  tail -n 40 /tmp/llama-pi/gemma-local.log
  sleep 10
done

echo '--- final log tail ---'
tail -n 120 /tmp/llama-pi/gemma-local.log

Server is up on 127.0.0.1:8080. I’m running one actual chat completion against the Gemma alias now.

The server works, but the first OpenAI chat response is garbage tokens (<unused25>...). I’m doing one more probe with simpler generation settings to see if this is just a chat-template issue.

>echo '--- /completion ---'
curl -sS http://127.0.0.1:8080/completion \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "Say exactly: llama.cpp works",
    "n_predict": 24,
    "temperature": 0,
    "stop": ["\n"]
  }'

echo '\n--- /v1/completions ---'
curl -sS http://127.0.0.1:8080/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gemma-4-26b-a4b-it-q4km",
    "prompt": "Say exactly: llama.cpp works",
    "max_tokens": 24,
    "temperature": 0
  }'

echo '\n--- recent server log ---'
tail -n 80 /tmp/llama-pi/gemma-local.log

Evaluating output issues

I need to provide a final, concise technical update about the server. The server is working, but it’s producing junk outputs, which might be an issue with either the model or the server itself. The user asked to try again, and we did. I should mention the command that was used and the current process ID and logs. It might be helpful to suggest the next steps, like trying a different Gemma quant/model or using the latest llama.cpp version, as there could be a bug in Gemma4.

Retried successfully with the local cached GGUFs instead of -hf.

Current state:

llama-server is running
listening on http://127.0.0.1:8080
/v1/models works

Launch command that worked:

bash

~/src/llama.cpp/build/bin/llama-server \  -m ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 \  --mmproj ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

~/src/llama.cpp/build/bin/llama-server \  -m ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 \  --mmproj ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 \  --alias gemma-4-26b-a4b-it-q4km \  --host 127.0.0.1 \  --port 8080

Verified:

model load completed
server bound port 8080
/v1/models returns gemma-4-26b-a4b-it-q4km

But generation is not usable yet:

/v1/chat/completions returns repeated <unused25>
/completion and /v1/completions do the same

So the server stack is working, but this Gemma 4 GGUF combo is not producing sane text in the current llama.cpp run. That points to one of:

a Gemma 4 support issue/regression in current llama.cpp
this specific GGUF needing different runtime flags
a model/template/tokenizer mismatch for this release

Files/logs:

server log: /tmp/llama-pi/gemma-local.log
pid file: /tmp/llama-pi/gemma-local.pid

If you want, next I can:

try alternate flags for Gemma 4 in llama.cpp
try a different Gemma 4 quant/repo
wire pi to the working endpoint anyway, then swap to a model that actually generates sane output

what does the model output?

It outputs garbage placeholder tokens.

Examples:

/v1/chat/completions

json

{  "message": {    "role": "assistant",    "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"  }}

{  "message": {    "role": "assistant",    "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"  }}

/completion

json

{  "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"}

{  "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"}

So:

server loads
API responds
actual text generation is broken for this model setup

Not a port/build issue anymore. It is a model/runtime correctness issue.

ok, delete llama cpp and the model cache shit

Removing:

~/src/llama.cpp
Hugging Face cache for ggml-org/gemma-4-26b-a4b-it-GGUF

I’ll stop any running llama-server first.

>set -e

PID_FILES="/tmp/llama-pi/gemma-server.pid /tmp/llama-pi/gemma-local.pid"
for pf in $PID_FILES; do
  if [ -f "$pf" ]; then
    pid=$(cat "$pf" 2>/dev/null || true)
    if [ -n "$pid" ] && ps -p "$pid" >/dev/null 2>&1; then
      kill "$pid" || true
      sleep 1
      if ps -p "$pid" >/dev/null 2>&1; then
        kill -9 "$pid" || true
      fi
    fi
  fi
done

rm -rf "$HOME/src/llama.cpp"
rm -rf "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF"
rm -rf /tmp/llama-pi

echo '--- remaining ---'
for p in "$HOME/src/llama.cpp" "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF" /tmp/llama-pi; do
  if [ -e "$p" ]; then
    echo "still exists: $p"
  else
    echo "removed: $p"
  fi
done

Removed:

~/src/llama.cpp
~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF
/tmp/llama-pi

Any launched llama-server processes from this session were stopped first.