Built and tested llama.cpp with Gemma model
Public
Built and tested llama.cpp with Gemma model
Agent
Pi
Model
GPT-5.4
Started
Apr 2, 7:22 PM
Messages103
Downloads0
Built llama.cpp from source.
BuildStarted llama-server with Gemma model.
SetupTested model output and found garbage tokens.
VerifyDeleted llama.cpp source and model cache.
Cleanup
/var/folders/49/l4171l3n219_xwq30fmyzr8w0000gn/T/pi-clipboard-0c15628d-b04b-4323-a156-17b60d3e00dc.png wanna try this out. figure out how to get the latest llama cpp going, ideally we compile from source.
Inspecting image for instructions
I need to check out the image the user mentioned, as they're trying to figure out how to get the latest LLaMA CPP compiled from source. I wonder if this image could have relevant details—I should inspect it carefully, but I don't want to modify anything unless the user asks me to. I’ll use the read tool to see what's in the image and discuss any steps if necessary. Let's take a look!
>rg -n "models.json|llama|localhost:8080|openai-completions|baseUrl" README.md packages/coding-agent/README.md packages/ai/README.md /opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/README.md /opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs -Spackages/coding-agent/README.md:120:**Custom providers & models:** Add providers via `~/.pi/agent/models.json` if they speak a supported API (OpenAI, Anthropic, Google). For custom APIs or OAuth, use extensions. See [docs/models.md](docs/models.md) and [docs/custom-provider.md](docs/custom-provider.md).
packages/ai/README.md:70:- **Any OpenAI-compatible API**: Ollama, vLLM, LM Studio, etc.
packages/ai/README.md:633:- **`openai-completions`**: OpenAI Chat Completions API (`streamOpenAICompletions`, `OpenAICompletionsOptions`)
packages/ai/README.md:732:- **xAI, Cerebras, Groq, etc.** models use the `openai-completions` API (OpenAI-compatible)
packages/ai/README.md:765:// Example: Ollama using OpenAI-compatible API
packages/ai/README.md:766:const ollamaModel: Model<'openai-completions'> = {
packages/ai/README.md:767: id: 'llama-3.1-8b',
packages/ai/README.md:768: name: 'Llama 3.1 8B (Ollama)',
packages/ai/README.md:769: api: 'openai-completions',
packages/ai/README.md:770: provider: 'ollama',
packages/ai/README.md:771: baseUrl: 'http://localhost:11434/v1',
packages/ai/README.md:780:const litellmModel: Model<'openai-completions'> = {
packages/ai/README.md:783: api: 'openai-completions',
packages/ai/README.md:785: baseUrl: 'http://localhost:4000/v1',
packages/ai/README.md:802: baseUrl: 'https://proxy.example.com/v1',
packages/ai/README.md:815:const response = await stream(ollamaModel, context, {
packages/ai/README.md:816: apiKey: 'dummy' // Ollama doesn't need a real key
packages/ai/README.md:822:This commonly applies to Ollama, vLLM, SGLang, and similar OpenAI-compatible servers. You can set `compat` at the provider level or per model.
packages/ai/README.md:825:const ollamaReasoningModel: Model<'openai-completions'> = {
packages/ai/README.md:827: name: 'GPT-OSS 20B (Ollama)',
packages/ai/README.md:828: api: 'openai-completions',
packages/ai/README.md:829: provider: 'ollama',
packages/ai/README.md:830: baseUrl: 'http://localhost:11434/v1',
packages/ai/README.md:845:The `openai-completions` API is implemented by many providers with minor differences. By default, the library auto-detects compatibility settings based on `baseUrl` for a small set of known OpenAI-compatible providers (Cerebras, xAI, Chutes, DeepSeek, zAi, OpenCode, etc.). For custom proxies or unknown endpoints, you can override these settings via the `compat` field. For `openai-responses` models, the compat field only supports Responses-specific flags.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/README.md:118:**Custom providers & models:** Add providers via `~/.pi/agent/models.json` if they speak a supported API (OpenAI, Anthropic, Google). For custom APIs or OAuth, use extensions. See [docs/models.md](docs/models.md) and [docs/custom-provider.md](docs/custom-provider.md).
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:37: // Override baseUrl for existing provider
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:39: baseUrl: "https://proxy.example.com"
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:44: baseUrl: "https://api.example.com",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:46: api: "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:69: baseUrl: "https://proxy.example.com"
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:79:// Both baseUrl and headers
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:81: baseUrl: "https://ai-gateway.corp.com/google",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:88:When only `baseUrl` and/or `headers` are provided (no `models`), all existing models for that provider are preserved with the new endpoint.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:96: baseUrl: "https://api.my-llm.com/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:98: api: "openai-completions", // which streaming API to use
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:127: baseUrl: "https://api.my-llm.com/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:129: api: "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:158:| `openai-completions` | OpenAI Chat Completions API and compatibles |
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:168:Most OpenAI-compatible providers work with `openai-completions`. Use `compat` for quirks:
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:193:> Migration note: Mistral moved from `openai-completions` to `mistral-conversations`.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:195:> If you intentionally route Mistral-compatible/custom endpoints through `openai-completions`, set `compat` flags explicitly as needed.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:203: baseUrl: "https://api.example.com",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:206: api: "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:219: baseUrl: "https://ai.corp.com/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:266: baseUrl: `https://${region}.ai.corp.com/v1`
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:311:- [openai-completions.ts](https://github.com/badlogic/pi-mono/blob/main/packages/ai/src/providers/openai-completions.ts) - OpenAI Chat Completions
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:476: baseUrl: "https://api.example.com",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:509: baseUrl?: string;
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/custom-provider.md:580: /** OpenAI compatibility settings for openai-completions API. */
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/sdk.md:279:- Custom models (`models.json`)
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/sdk.md:298:// Find any model by provider/id, including custom models from models.json
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/sdk.md:333:4. Fallback resolver (for custom provider keys from `models.json`)
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/sdk.md:338:// Default: uses ~/.pi/agent/auth.json and ~/.pi/agent/models.json
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/sdk.md:353:const customRegistry = ModelRegistry.create(customAuth, "/my/app/models.json");
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/sdk.md:361:// No custom models.json (built-in models only)
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/sdk.md:790:// Model registry (no custom models.json)
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:3:Add custom providers and models (Ollama, vLLM, LM Studio, proxies) via `~/.pi/agent/models.json`.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:18:For local models (Ollama, LM Studio, vLLM), only `id` is required per model:
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:23: "ollama": {
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:24: "baseUrl": "http://localhost:11434/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:25: "api": "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:26: "apiKey": "ollama",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:28: { "id": "llama3.1:8b" },
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:36:The `apiKey` is required but Ollama ignores it, so any value works.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:40:You can set `compat` at the provider level to apply to all models, or at the model level to override a specific model. This commonly applies to Ollama, vLLM, SGLang, and similar OpenAI-compatible servers.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:45: "ollama": {
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:46: "baseUrl": "http://localhost:11434/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:47: "api": "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:48: "apiKey": "ollama",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:71: "ollama": {
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:72: "baseUrl": "http://localhost:11434/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:73: "api": "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:74: "apiKey": "ollama",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:77: "id": "llama3.1:8b",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:97:| `openai-completions` | OpenAI Chat Completions (most compatible) |
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:108:| `baseUrl` | API endpoint URL |
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:134:For `models.json`, shell commands are resolved at request time. pi intentionally does not apply built-in TTL, stale reuse, or recovery logic for arbitrary commands. Different commands need different caching and failure strategies, and pi cannot infer the right one.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:146: "baseUrl": "https://proxy.example.com/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:185: "baseUrl": "https://my-proxy.example.com/v1"
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:199: "baseUrl": "https://my-proxy.example.com/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:242:- You can combine provider-level `baseUrl`/`headers` with `modelOverrides`.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:256: "baseUrl": "http://localhost:8080/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:257: "api": "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:292: "baseUrl": "https://openrouter.ai/api/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:294: "api": "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:318: "baseUrl": "https://ai-gateway.vercel.sh/v1",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/models.md:320: "api": "openai-completions",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/extensions.md:1316: baseUrl: "https://proxy.example.com",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/extensions.md:1332:// Override baseUrl for an existing provider (keeps all models)
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/extensions.md:1334: baseUrl: "https://proxy.example.com"
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/extensions.md:1339: baseUrl: "https://ai.corp.com",
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/extensions.md:1362:- `baseUrl` - API endpoint URL. Required when defining models.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/extensions.md:1364:- `api` - API type: `"anthropic-messages"`, `"openai-completions"`, `"openai-responses"`, etc.
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/providers.md:184:**Via models.json:** Add Ollama, LM Studio, vLLM, or any provider that speaks a supported API (OpenAI Completions, OpenAI Responses, Anthropic Messages, Google Generative AI). See [models.md](models.md).
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/providers.md:195:4. Custom provider keys from `models.json`
/opt/homebrew/lib/node_modules/@mariozechner/pi-coding-agent/docs/rpc.md:1186: "baseUrl": "https://api.anthropic.com",Exploring documentation needs
I’m thinking I might need to read through the docs, especially models.md and the README file, since they seem important for the topic related to pi. I wonder if just reading models.json usage will be sufficient, or if I should dive into the custom-provider details too. To be on the safe side, I'll check both models.md and the coding-agent README fully. It seems like a thorough approach!
Considering Llama CPP Setup
I think we have enough documentation to consider compiling from source with Metal/CUDA, especially since the user is likely on a Mac. I should ask if they want help with installation or building. There wasn’t a specific request for implementation, but it seems like they want assistance setting it up, so I could provide the steps and relevant commands. I might also want to include an example, including the screenshot command and mention the model ID for the llama-server.
Reviewing Llama CPP Compilation
I’m considering that in the OpenAI endpoint, the model ID might relate to a repository. It could be helpful to verify this with the llama.cpp documentation, though I might not have internet access. I could check what’s already installed, potentially using commands like git ls-remote. For macOS, I want to outline the Metal build commands. I think I should include commands like git clone, cmake settings, and the generated binary path. I’ll also mention the correct model ID in the models.json.
Setting Up Model Aliases
I just realized that the OpenAI requests require a model field, and the llama-server can use a --alias argument to set the model name, which is really helpful! I should recommend running the server with --alias gemma-4-26b-a4b-it-q4km, and then that ID should go into the models.json. I’m also thinking it might be good to mention using --jinja for template purposes as well. This feels like a solid plan to provide for the user!
Preparing model setup instructions
I need to mention a few key steps for setting up the model. First, clone and build the project, then run the server with the specified command. Testing can be done through curl commands to check models and completions. I should include installation of dependencies like CMake and Xcode CLI tools. If users want GPU support on NVIDIA or AMD, I’ll explain the necessary options, but generally, they may be on Mac. Maybe I should also clarify if anyone wants help with the compilation process.
Yep. The screenshot flow is plausible, but I would tighten it a bit so pi can target a stable model ID.
For macOS, this is the clean path to get the latest llama.cpp from source and expose it to pi over the OpenAI-compatible server.
Requirements:
xcode-select --installbrew install cmakeThen:
cd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -jBinary should end up at:
~/src/llama.cpp/build/bin/llama-serverFor Apple Silicon, GGML_METAL=ON is the important part.
Your screenshot uses:
llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_MI would run it like this instead:
~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080Why --alias matters:
model string to the OpenAI-compatible endpointmodels.json can use that exact aliasIf you want the server reachable from other machines on your LAN, use --host 0.0.0.0 instead.
Check that the OpenAI-compatible API is up:
curl http://127.0.0.1:8080/v1/modelsYou should see the alias in the response.
Then a quick chat test:
curl http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "gemma-4-26b-a4b-it-q4km", "messages": [ { "role": "user", "content": "say hello" } ], "stream": false }'If that works, pi can use it.
Create ~/.pi/agent/models.json:
{ "providers": { "llama-cpp": { "baseUrl": "http://127.0.0.1:8080/v1", "api": "openai-completions", "apiKey": "dummy", "compat": { "supportsDeveloperRole": false }, "models": [ { "id": "gemma-4-26b-a4b-it-q4km", "name": "Gemma 4 26B A4B IT Q4_K_M (llama.cpp local)", "reasoning": false, "input": ["text"] } ] } }}Notes:
apiKey is required by pi config shape, but local llama-server does not caresupportsDeveloperRole: false is a good default for OpenAI-compatible local serversinput to ["text", "image"]Then launch pi and select it with /model, or directly:
pi --model llama-cpp/gemma-4-26b-a4b-it-q4kmIf requests fail, the usual compatibility knobs for local OpenAI-compatible servers are:
"compat": { "supportsDeveloperRole": false, "supportsReasoningEffort": false, "supportsUsageInStreaming": false, "maxTokensField": "max_tokens"}I would start with only supportsDeveloperRole: false, then add others only if needed.
cd ~/src/llama.cppgit pullcmake -B build -DGGML_METAL=ONcmake --build build --config Release -jYou can swap the model source without changing pi config, as long as you keep the same alias:
~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q5_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080Though in practice I would rename the alias to match the quant.
For regular use, run it in tmux:
tmux new -s llama~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080If you want the shortest path, use exactly this:
xcode-select --installbrew install cmakecd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080{ "providers": { "llama-cpp": { "baseUrl": "http://127.0.0.1:8080/v1", "api": "openai-completions", "apiKey": "dummy", "compat": { "supportsDeveloperRole": false }, "models": [ { "id": "gemma-4-26b-a4b-it-q4km" } ] } }}If you want, I can next turn this into:
models.json for a few candidate local models, not just Gemma 4.ok, build llama first, and test it works with the gemma model
Planning implementation steps
I need to implement a concrete task, and I should check if there’s a README for guidance. I might consider using the bash command for tasks like ls, and I should check if the /src directory exists. Compiling the source could take some time and require internet, so I need to be cautious about permissions and possibly avoid using sudo. I might also need to see if cmake is already installed. Let’s go through these steps with clear commentary.
Building llama.cpp, then I’ll start llama-server with the Gemma GGUF and verify the OpenAI-compatible endpoint.
>pwd && uname -a && command -v cmake && cmake --version && command -v git && xcode-select -p && ls -d ~/src || true/Users/badlogic/workspaces/pi-mono
Darwin Marios-MacBook-Pro.local 25.1.0 Darwin Kernel Version 25.1.0: Mon Oct 20 19:32:41 PDT 2025; root:xnu-12377.41.6~2/RELEASE_ARM64_T6000 arm64
/opt/homebrew/bin/cmake
cmake version 4.0.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
/usr/bin/git
/Applications/Xcode.app/Contents/Developer
ls: /Users/badlogic/src: No such file or directory>mkdir -p ~/src && cd ~/src && if [ ! -d llama.cpp/.git ]; then git clone https://github.com/ggml-org/llama.cpp.git; fi && cd llama.cpp && git pull --ff-only && cmake -B build -DGGML_METAL=ON && cmake --build build --config Release -j8Cloning into 'llama.cpp'...
Already up to date.
-- The C compiler identification is AppleClang 17.0.0.17000404
-- The CXX compiler identification is AppleClang 17.0.0.17000404
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_BUILD_TYPE=Release
-- Found Git: /usr/bin/git (found version "2.50.1 (Apple Git-155)")
-- The ASM compiler identification is AppleClang
-- Found assembler: /usr/bin/cc
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: arm64
-- GGML_SYSTEM_ARCH: ARM
-- Including CPU backend
-- Accelerate framework found
-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
CMake Warning at ggml/src/ggml-cpu/CMakeLists.txt:84 (message):
OpenMP not found
Call Stack (most recent call first):
ggml/src/CMakeLists.txt:445 (ggml_add_cpu_backend_variant_impl)
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
CMake Warning at ggml/src/ggml-cpu/CMakeLists.txt:146 (message):
ARM -march/-mcpu not found, -mcpu=native will be used
Call Stack (most recent call first):
ggml/src/CMakeLists.txt:445 (ggml_add_cpu_backend_variant_impl)
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod - Success
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm - Success
-- Performing Test GGML_MACHINE_SUPPORTS_sve
-- Performing Test GGML_MACHINE_SUPPORTS_sve - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosve
-- Performing Test GGML_MACHINE_SUPPORTS_nosve - Success
-- Performing Test GGML_MACHINE_SUPPORTS_sme
-- Performing Test GGML_MACHINE_SUPPORTS_sme - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosme
-- Performing Test GGML_MACHINE_SUPPORTS_nosme - Success
-- Checking for ARM features using flags:
-- -U__ARM_FEATURE_MATMUL_INT8
-- -U__ARM_FEATURE_SVE
-- -U__ARM_FEATURE_SME
-- -mcpu=native+dotprod+noi8mm+nosve+nosme
-- Performing Test HAVE_DOTPROD
-- Performing Test HAVE_DOTPROD - Success
-- Performing Test HAVE_SVE
-- Performing Test HAVE_SVE - Failed
-- Performing Test HAVE_MATMUL_INT8
-- Performing Test HAVE_MATMUL_INT8 - Failed
-- Performing Test HAVE_FMA
-- Performing Test HAVE_FMA - Success
-- Performing Test HAVE_FP16_VECTOR_ARITHMETIC
-- Performing Test HAVE_FP16_VECTOR_ARITHMETIC - Success
-- Performing Test HAVE_SME
-- Performing Test HAVE_SME - Failed
-- Adding CPU backend variant ggml-cpu: -U__ARM_FEATURE_MATMUL_INT8;-U__ARM_FEATURE_SVE;-U__ARM_FEATURE_SME;-mcpu=native+dotprod+noi8mm+nosve+nosme
-- Looking for dgemm_
-- Looking for dgemm_ - found
-- Found BLAS: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework
-- BLAS found, Libraries: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework
-- BLAS found, Includes:
-- Including BLAS backend
-- Metal framework found
-- Including METAL backend
-- ggml version: 0.9.11
-- ggml commit: a1cfb6453
-- Found OpenSSL: /opt/homebrew/Cellar/openssl@3/3.6.0/lib/libcrypto.dylib (found version "3.6.0")
-- Performing Test OPENSSL_VERSION_SUPPORTED
-- Performing Test OPENSSL_VERSION_SUPPORTED - Success
-- OpenSSL found: 3.6.0
-- Generating embedded license file for target: common
-- Configuring done (7.9s)
-- Generating done (0.8s)
-- Build files have been written to: /Users/badlogic/src/llama.cpp/build
[ 0%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[ 0%] Building C object examples/gguf-hash/CMakeFiles/sha256.dir/deps/sha256/sha256.c.o
[ 0%] Building CXX object common/CMakeFiles/build_info.dir/build-info.cpp.o
[ 0%] Building CXX object vendor/cpp-httplib/CMakeFiles/cpp-httplib.dir/httplib.cpp.o
[ 0%] Building CXX object tools/mtmd/CMakeFiles/llama-gemma3-cli.dir/deprecation-warning.cpp.o
[ 0%] Building CXX object tools/mtmd/CMakeFiles/llama-llava-cli.dir/deprecation-warning.cpp.o
[ 0%] Building C object examples/gguf-hash/CMakeFiles/sha1.dir/deps/sha1/sha1.c.o
[ 1%] Building C object examples/gguf-hash/CMakeFiles/xxhash.dir/deps/xxhash/xxhash.c.o
[ 1%] Built target build_info
[ 2%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml.cpp.o
[ 2%] Built target sha256
[ 2%] Built target sha1
[ 2%] Building CXX object tools/mtmd/CMakeFiles/llama-qwen2vl-cli.dir/deprecation-warning.cpp.o
[ 2%] Building CXX object tools/mtmd/CMakeFiles/llama-minicpmv-cli.dir/deprecation-warning.cpp.o
[ 2%] Built target xxhash
[ 2%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[ 3%] Linking CXX executable ../../bin/llama-gemma3-cli
[ 4%] Linking CXX executable ../../bin/llama-llava-cli
[ 4%] Built target llama-llava-cli
[ 4%] Built target llama-gemma3-cli
[ 4%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[ 4%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
[ 4%] Linking CXX executable ../../bin/llama-minicpmv-cli
[ 5%] Linking CXX executable ../../bin/llama-qwen2vl-cli
[ 5%] Built target llama-minicpmv-cli
[ 5%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
[ 5%] Built target llama-qwen2vl-cli
[ 6%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[ 6%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
[ 6%] Linking CXX shared library ../../bin/libggml-base.dylib
[ 6%] Built target ggml-base
[ 6%] Generate assembly for embedded Metal library
Embedding Metal library
[ 6%] Building CXX object ggml/src/ggml-blas/CMakeFiles/ggml-blas.dir/ggml-blas.cpp.o
[ 6%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o
[ 6%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/repack.cpp.o
[ 7%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o
[ 8%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/hbm.cpp.o
[ 8%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o
[ 8%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/traits.cpp.o
[ 8%] Building CXX object ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal.cpp.o
[ 8%] Building C object ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal-device.m.o
[ 8%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/amx.cpp.o
[ 8%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/mmq.cpp.o
[ 8%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/binary-ops.cpp.o
[ 9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/unary-ops.cpp.o
[ 9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/vec.cpp.o
[ 9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ops.cpp.o
[ 10%] Building CXX object ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal-device.cpp.o
[ 10%] Linking CXX shared library ../../../bin/libggml-blas.dylib
[ 10%] Built target ggml-blas
[ 10%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/llamafile/sgemm.cpp.o
[ 10%] Building CXX object ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal-common.cpp.o
[ 10%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/arm/quants.c.o
[ 11%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/arm/repack.cpp.o
[ 11%] Building C object ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal-context.m.o
[ 11%] Building CXX object ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal-ops.cpp.o
[ 12%] Building ASM object ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/autogenerated/ggml-metal-embed.s.o
[ 12%] Linking CXX shared library ../../../bin/libggml-metal.dylib
[ 12%] Built target ggml-metal
[ 12%] Linking CXX static library libcpp-httplib.a
[ 12%] Built target cpp-httplib
[ 12%] Linking CXX shared library ../../bin/libggml-cpu.dylib
[ 12%] Built target ggml-cpu
[ 12%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-backend-reg.cpp.o
[ 12%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-backend-dl.cpp.o
[ 12%] Linking CXX shared library ../../bin/libggml.dylib
[ 12%] Built target ggml
[ 12%] Building CXX object examples/gguf/CMakeFiles/llama-gguf.dir/gguf.cpp.o
[ 12%] Building CXX object examples/gguf-hash/CMakeFiles/llama-gguf-hash.dir/gguf-hash.cpp.o
[ 12%] Building CXX object src/CMakeFiles/llama.dir/llama.cpp.o
[ 12%] Building CXX object src/CMakeFiles/llama.dir/llama-batch.cpp.o
[ 12%] Building CXX object src/CMakeFiles/llama.dir/llama-chat.cpp.o
[ 12%] Building CXX object src/CMakeFiles/llama.dir/llama-arch.cpp.o
[ 13%] Building CXX object src/CMakeFiles/llama.dir/llama-adapter.cpp.o
[ 13%] Building CXX object src/CMakeFiles/llama.dir/llama-context.cpp.o
[ 14%] Linking CXX executable ../../bin/llama-gguf
[ 14%] Built target llama-gguf
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-cparams.cpp.o
[ 15%] Linking CXX executable ../../bin/llama-gguf-hash
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-grammar.cpp.o
[ 15%] Built target llama-gguf-hash
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-graph.cpp.o
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-hparams.cpp.o
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-impl.cpp.o
[ 16%] Building CXX object src/CMakeFiles/llama.dir/llama-io.cpp.o
[ 16%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache.cpp.o
[ 16%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache-iswa.cpp.o
[ 16%] Building CXX object src/CMakeFiles/llama.dir/llama-memory.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-hybrid.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-hybrid-iswa.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-recurrent.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-mmap.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-model-loader.cpp.o
[ 18%] Building CXX object src/CMakeFiles/llama.dir/llama-model-saver.cpp.o
[ 18%] Building CXX object src/CMakeFiles/llama.dir/llama-model.cpp.o
[ 18%] Building CXX object src/CMakeFiles/llama.dir/llama-quant.cpp.o
[ 18%] Building CXX object src/CMakeFiles/llama.dir/llama-sampler.cpp.o
[ 18%] Building CXX object src/CMakeFiles/llama.dir/llama-vocab.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/unicode-data.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/unicode.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/models/afmoe.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/models/apertus.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/models/arcee.cpp.o
[ 20%] Building CXX object src/CMakeFiles/llama.dir/models/arctic.cpp.o
[ 20%] Building CXX object src/CMakeFiles/llama.dir/models/arwkv7.cpp.o
[ 20%] Building CXX object src/CMakeFiles/llama.dir/models/baichuan.cpp.o
[ 20%] Building CXX object src/CMakeFiles/llama.dir/models/bailingmoe.cpp.o
[ 21%] Building CXX object src/CMakeFiles/llama.dir/models/bailingmoe2.cpp.o
[ 21%] Building CXX object src/CMakeFiles/llama.dir/models/bert.cpp.o
[ 21%] Building CXX object src/CMakeFiles/llama.dir/models/bitnet.cpp.o
[ 21%] Building CXX object src/CMakeFiles/llama.dir/models/bloom.cpp.o
[ 21%] Building CXX object src/CMakeFiles/llama.dir/models/chameleon.cpp.o
[ 22%] Building CXX object src/CMakeFiles/llama.dir/models/chatglm.cpp.o
[ 22%] Building CXX object src/CMakeFiles/llama.dir/models/codeshell.cpp.o
[ 22%] Building CXX object src/CMakeFiles/llama.dir/models/cogvlm.cpp.o
[ 22%] Building CXX object src/CMakeFiles/llama.dir/models/cohere2-iswa.cpp.o
[ 22%] Building CXX object src/CMakeFiles/llama.dir/models/command-r.cpp.o
[ 23%] Building CXX object src/CMakeFiles/llama.dir/models/dbrx.cpp.o
[ 23%] Building CXX object src/CMakeFiles/llama.dir/models/deci.cpp.o
[ 23%] Building CXX object src/CMakeFiles/llama.dir/models/deepseek.cpp.o
[ 23%] Building CXX object src/CMakeFiles/llama.dir/models/deepseek2.cpp.o
[ 24%] Building CXX object src/CMakeFiles/llama.dir/models/delta-net-base.cpp.o
[ 24%] Building CXX object src/CMakeFiles/llama.dir/models/dots1.cpp.o
[ 24%] Building CXX object src/CMakeFiles/llama.dir/models/dream.cpp.o
[ 24%] Building CXX object src/CMakeFiles/llama.dir/models/ernie4-5-moe.cpp.o
[ 24%] Building CXX object src/CMakeFiles/llama.dir/models/ernie4-5.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/eurobert.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/exaone-moe.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/exaone.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/exaone4.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/falcon-h1.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/falcon.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/gemma-embedding.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/gemma.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/gemma2-iswa.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/gemma3.cpp.o
[ 27%] Building CXX object src/CMakeFiles/llama.dir/models/gemma3n-iswa.cpp.o
[ 27%] Building CXX object src/CMakeFiles/llama.dir/models/gemma4-iswa.cpp.o
[ 27%] Building CXX object src/CMakeFiles/llama.dir/models/glm4-moe.cpp.o
[ 27%] Building CXX object src/CMakeFiles/llama.dir/models/glm4.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/models/gpt2.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/models/gptneox.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/models/granite-hybrid.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/models/granite.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/models/grok.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/models/grovemoe.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/models/hunyuan-dense.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/models/hunyuan-moe.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/models/internlm2.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/models/jais.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/jais2.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/jamba.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/kimi-linear.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/lfm2.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/llada-moe.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/models/llada.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/models/llama-iswa.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/models/llama.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/models/maincoder.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/mamba-base.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/mamba.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/mimo2-iswa.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/minicpm3.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/minimax-m2.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/mistral3.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/modern-bert.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/mpt.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/nemotron-h.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/nemotron.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/neo-bert.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/olmo.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/olmo2.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/olmoe.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/openai-moe-iswa.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/openelm.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/orion.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/paddleocr.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/pangu-embedded.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/phi2.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/phi3.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/plamo.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/plamo2.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/plamo3.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/plm.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/qwen.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2moe.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2vl.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/qwen35.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/qwen35moe.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3moe.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3next.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3vl-moe.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3vl.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/refact.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/rnd1.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6-base.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6qwen2.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv7-base.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv7.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/seed-oss.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/smallthinker.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/smollm3.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/stablelm.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/starcoder.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/starcoder2.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/step35-iswa.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/t5-dec.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/t5-enc.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/wavtokenizer-dec.cpp.o
[ 43%] Building CXX object src/CMakeFiles/llama.dir/models/xverse.cpp.o
[ 43%] Linking CXX shared library ../bin/libllama.dylib
[ 43%] Built target llama
[ 44%] Building CXX object examples/simple-chat/CMakeFiles/llama-simple-chat.dir/simple-chat.cpp.o
[ 44%] Building C object tests/CMakeFiles/test-c.dir/test-c.c.o
[ 44%] Building CXX object examples/simple/CMakeFiles/llama-simple.dir/simple.cpp.o
[ 44%] Building CXX object common/CMakeFiles/common.dir/arg.cpp.o
[ 44%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-audio.cpp.o
[ 44%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd.cpp.o
[ 44%] Building CXX object common/CMakeFiles/common.dir/chat-auto-parser-helpers.cpp.o
[ 44%] Building CXX object common/CMakeFiles/common.dir/chat-auto-parser-generator.cpp.o
[ 44%] Linking C executable ../bin/test-c
[ 44%] Built target test-c
[ 44%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-image.cpp.o
[ 44%] Linking CXX executable ../../bin/llama-simple
[ 44%] Built target llama-simple
[ 45%] Building CXX object common/CMakeFiles/common.dir/chat-diff-analyzer.cpp.o
[ 45%] Linking CXX executable ../../bin/llama-simple-chat
[ 45%] Built target llama-simple-chat
[ 46%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-helper.cpp.o
[ 46%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/clip.cpp.o
[ 46%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/cogvlm.cpp.o
[ 46%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/conformer.cpp.o
[ 46%] Building CXX object common/CMakeFiles/common.dir/chat-peg-parser.cpp.o
[ 47%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/gemma4v.cpp.o
[ 47%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/glm4v.cpp.o
[ 47%] Building CXX object common/CMakeFiles/common.dir/chat.cpp.o
[ 47%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/internvl.cpp.o
[ 47%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/kimivl.cpp.o
[ 47%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/kimik25.cpp.o
[ 48%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/nemotron-v2-vl.cpp.o
[ 48%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/llama4.cpp.o
[ 48%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/llava.cpp.o
[ 48%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/minicpmv.cpp.o
[ 48%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/paddleocr.cpp.o
[ 49%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/pixtral.cpp.o
[ 49%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/qwen2vl.cpp.o
[ 49%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/qwen3vl.cpp.o
[ 49%] Building CXX object common/CMakeFiles/common.dir/common.cpp.o
[ 49%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/siglip.cpp.o
[ 49%] Building CXX object common/CMakeFiles/common.dir/console.cpp.o
[ 49%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/whisper-enc.cpp.o
[ 50%] Building CXX object common/CMakeFiles/common.dir/debug.cpp.o
[ 51%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/deepseekocr.cpp.o
[ 51%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/mobilenetv5.cpp.o
[ 51%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/youtuvl.cpp.o
[ 51%] Building CXX object common/CMakeFiles/common.dir/download.cpp.o
[ 51%] Building CXX object common/CMakeFiles/common.dir/hf-cache.cpp.o
[ 51%] Building CXX object common/CMakeFiles/common.dir/json-partial.cpp.o
[ 51%] Building CXX object common/CMakeFiles/common.dir/json-schema-to-grammar.cpp.o
[ 51%] Linking CXX shared library ../../bin/libmtmd.dylib
[ 51%] Built target mtmd
[ 52%] Building CXX object common/CMakeFiles/common.dir/llguidance.cpp.o
[ 52%] Building CXX object common/CMakeFiles/common.dir/log.cpp.o
[ 52%] Building CXX object common/CMakeFiles/common.dir/ngram-cache.cpp.o
[ 52%] Building CXX object common/CMakeFiles/common.dir/ngram-map.cpp.o
[ 53%] Building CXX object common/CMakeFiles/common.dir/ngram-mod.cpp.o
[ 53%] Building CXX object common/CMakeFiles/common.dir/peg-parser.cpp.o
[ 53%] Building CXX object common/CMakeFiles/common.dir/preset.cpp.o
[ 53%] Building CXX object common/CMakeFiles/common.dir/regex-partial.cpp.o
[ 53%] Building CXX object common/CMakeFiles/common.dir/reasoning-budget.cpp.o
[ 54%] Building CXX object common/CMakeFiles/common.dir/sampling.cpp.o
[ 54%] Building CXX object common/CMakeFiles/common.dir/speculative.cpp.o
[ 54%] Building CXX object common/CMakeFiles/common.dir/unicode.cpp.o
[ 54%] Building CXX object common/CMakeFiles/common.dir/jinja/lexer.cpp.o
[ 54%] Building CXX object common/CMakeFiles/common.dir/jinja/parser.cpp.o
[ 55%] Building CXX object common/CMakeFiles/common.dir/jinja/runtime.cpp.o
[ 55%] Building CXX object common/CMakeFiles/common.dir/jinja/value.cpp.o
[ 55%] Building CXX object common/CMakeFiles/common.dir/jinja/string.cpp.o
[ 55%] Building CXX object common/CMakeFiles/common.dir/jinja/caps.cpp.o
[ 55%] Building CXX object common/CMakeFiles/common.dir/__/license.cpp.o
[ 56%] Linking CXX static library libcommon.a
[ 56%] Built target common
[ 56%] Building CXX object tests/CMakeFiles/test-tokenizer-0.dir/test-tokenizer-0.cpp.o
[ 56%] Building CXX object tests/CMakeFiles/test-sampling.dir/test-sampling.cpp.o
[ 57%] Building CXX object tests/CMakeFiles/test-grammar-integration.dir/test-grammar-integration.cpp.o
[ 57%] Building CXX object tests/CMakeFiles/test-json-schema-to-grammar.dir/test-json-schema-to-grammar.cpp.o
[ 57%] Building CXX object tests/CMakeFiles/test-chat.dir/test-chat.cpp.o
[ 57%] Building CXX object tests/CMakeFiles/test-reasoning-budget.dir/test-reasoning-budget.cpp.o
[ 57%] Building CXX object tests/CMakeFiles/test-llama-grammar.dir/test-llama-grammar.cpp.o
[ 57%] Building CXX object tests/CMakeFiles/test-grammar-parser.dir/test-grammar-parser.cpp.o
[ 58%] Building CXX object tests/CMakeFiles/test-reasoning-budget.dir/get-model.cpp.o
[ 58%] Linking CXX executable ../bin/test-reasoning-budget
[ 59%] Building CXX object tests/CMakeFiles/test-sampling.dir/get-model.cpp.o
[ 59%] Linking CXX executable ../bin/test-sampling
[ 59%] Built target test-reasoning-budget
[ 59%] Building CXX object tests/CMakeFiles/test-quantize-stats.dir/test-quantize-stats.cpp.o
[ 60%] Building CXX object tests/CMakeFiles/test-grammar-parser.dir/get-model.cpp.o
[ 60%] Linking CXX executable ../bin/test-grammar-parser
[ 60%] Built target test-sampling
[ 60%] Building CXX object tests/CMakeFiles/test-gbnf-validator.dir/test-gbnf-validator.cpp.o
[ 60%] Built target test-grammar-parser
[ 60%] Linking CXX executable ../bin/test-tokenizer-0
[ 61%] Building CXX object tests/CMakeFiles/test-llama-grammar.dir/get-model.cpp.o
[ 62%] Building CXX object tests/CMakeFiles/test-tokenizer-1-bpe.dir/test-tokenizer-1-bpe.cpp.o
[ 62%] Linking CXX executable ../bin/test-llama-grammar
[ 62%] Built target test-tokenizer-0
[ 62%] Building CXX object tests/CMakeFiles/test-tokenizer-1-spm.dir/test-tokenizer-1-spm.cpp.o
[ 62%] Built target test-llama-grammar
[ 62%] Building CXX object tests/CMakeFiles/test-llama-archs.dir/test-llama-archs.cpp.o
[ 62%] Linking CXX executable ../bin/test-gbnf-validator
[ 62%] Linking CXX executable ../bin/test-tokenizer-1-bpe
[ 62%] Built target test-gbnf-validator
[ 62%] Building CXX object tests/CMakeFiles/test-chat-peg-parser.dir/test-chat-peg-parser.cpp.o
[ 62%] Built target test-tokenizer-1-bpe
[ 62%] Linking CXX executable ../bin/test-tokenizer-1-spm
[ 62%] Building CXX object tests/CMakeFiles/test-jinja.dir/test-jinja.cpp.o
[ 62%] Built target test-tokenizer-1-spm
[ 62%] Building CXX object tests/CMakeFiles/test-chat-auto-parser.dir/test-chat-auto-parser.cpp.o
[ 62%] Building CXX object tests/CMakeFiles/test-llama-archs.dir/get-model.cpp.o
[ 62%] Linking CXX executable ../bin/test-llama-archs
[ 62%] Built target test-llama-archs
[ 63%] Building CXX object tests/CMakeFiles/test-chat-template.dir/test-chat-template.cpp.o
[ 63%] Linking CXX executable ../bin/test-quantize-stats
[ 63%] Building CXX object tests/CMakeFiles/test-grammar-integration.dir/get-model.cpp.o
[ 63%] Built target test-quantize-stats
[ 63%] Linking CXX executable ../bin/test-grammar-integration
[ 64%] Building CXX object tests/CMakeFiles/test-json-partial.dir/test-json-partial.cpp.o
[ 64%] Building CXX object tests/CMakeFiles/test-json-schema-to-grammar.dir/get-model.cpp.o
[ 64%] Built target test-grammar-integration
[ 64%] Building CXX object tests/CMakeFiles/test-log.dir/test-log.cpp.o
[ 65%] Linking CXX executable ../bin/test-json-schema-to-grammar
[ 65%] Built target test-json-schema-to-grammar
[ 66%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/test-peg-parser.cpp.o
[ 66%] Building CXX object tests/CMakeFiles/test-log.dir/get-model.cpp.o
[ 67%] Linking CXX executable ../bin/test-log
[ 67%] Built target test-log
[ 67%] Building CXX object tests/CMakeFiles/test-regex-partial.dir/test-regex-partial.cpp.o
[ 67%] Building CXX object tests/CMakeFiles/test-json-partial.dir/get-model.cpp.o
[ 67%] Linking CXX executable ../bin/test-json-partial
[ 67%] Built target test-json-partial
[ 68%] Building CXX object tests/CMakeFiles/test-thread-safety.dir/test-thread-safety.cpp.o
[ 68%] Building CXX object tests/CMakeFiles/test-regex-partial.dir/get-model.cpp.o
[ 68%] Linking CXX executable ../bin/test-regex-partial
[ 68%] Built target test-regex-partial
[ 68%] Building CXX object tests/CMakeFiles/test-arg-parser.dir/test-arg-parser.cpp.o
[ 68%] Building CXX object tests/CMakeFiles/test-thread-safety.dir/get-model.cpp.o
[ 68%] Building CXX object tests/CMakeFiles/test-chat-template.dir/get-model.cpp.o
[ 68%] Linking CXX executable ../bin/test-thread-safety
[ 68%] Linking CXX executable ../bin/test-chat-template
[ 68%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/simple-tokenize.cpp.o
[ 68%] Built target test-thread-safety
[ 68%] Built target test-chat-template
[ 68%] Building CXX object tests/CMakeFiles/test-opt.dir/test-opt.cpp.o
[ 69%] Building CXX object tests/CMakeFiles/test-gguf.dir/test-gguf.cpp.o
[ 69%] Building CXX object tests/CMakeFiles/test-arg-parser.dir/get-model.cpp.o
[ 69%] Linking CXX executable ../bin/test-arg-parser
[ 69%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-basic.cpp.o
[ 69%] Built target test-arg-parser
[ 69%] Building CXX object tests/CMakeFiles/test-backend-ops.dir/test-backend-ops.cpp.o
[ 69%] Building CXX object tests/CMakeFiles/test-opt.dir/get-model.cpp.o
[ 69%] Linking CXX executable ../bin/test-opt
[ 69%] Built target test-opt
[ 69%] Building CXX object tests/CMakeFiles/test-model-load-cancel.dir/test-model-load-cancel.cpp.o
[ 69%] Building CXX object tests/CMakeFiles/test-gguf.dir/get-model.cpp.o
[ 69%] Linking CXX executable ../bin/test-gguf
[ 69%] Building CXX object tests/CMakeFiles/test-model-load-cancel.dir/get-model.cpp.o
[ 69%] Built target test-gguf
[ 70%] Building CXX object tests/CMakeFiles/test-chat.dir/get-model.cpp.o
[ 70%] Linking CXX executable ../bin/test-model-load-cancel
[ 71%] Building CXX object tests/CMakeFiles/test-autorelease.dir/test-autorelease.cpp.o
[ 71%] Built target test-model-load-cancel
[ 71%] Building CXX object tests/CMakeFiles/test-backend-sampler.dir/test-backend-sampler.cpp.o
[ 71%] Linking CXX executable ../bin/test-chat
[ 71%] Built target test-chat
[ 71%] Building CXX object tests/CMakeFiles/test-state-restore-fragmented.dir/test-state-restore-fragmented.cpp.o
[ 71%] Building CXX object tests/CMakeFiles/test-autorelease.dir/get-model.cpp.o
[ 71%] Linking CXX executable ../bin/test-autorelease
[ 71%] Built target test-autorelease
[ 71%] Building CXX object tests/CMakeFiles/test-chat-peg-parser.dir/peg-parser/simple-tokenize.cpp.o
[ 71%] Building CXX object tests/CMakeFiles/test-jinja.dir/get-model.cpp.o
[ 71%] Building CXX object tests/CMakeFiles/test-chat-peg-parser.dir/get-model.cpp.o
[ 71%] Linking CXX executable ../bin/test-jinja
[ 71%] Building CXX object tests/CMakeFiles/test-barrier.dir/test-barrier.cpp.o
[ 71%] Building CXX object tests/CMakeFiles/test-chat-auto-parser.dir/get-model.cpp.o
[ 71%] Built target test-jinja
[ 71%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-gbnf-generation.cpp.o
[ 72%] Linking CXX executable ../bin/test-chat-auto-parser
[ 72%] Built target test-chat-auto-parser
[ 72%] Linking CXX executable ../bin/test-chat-peg-parser
[ 72%] Building CXX object tests/CMakeFiles/test-quantize-fns.dir/test-quantize-fns.cpp.o
[ 72%] Built target test-chat-peg-parser
[ 72%] Building CXX object tests/CMakeFiles/test-quantize-perf.dir/test-quantize-perf.cpp.o
[ 72%] Building CXX object tests/CMakeFiles/test-backend-sampler.dir/get-model.cpp.o
[ 72%] Linking CXX executable ../bin/test-backend-sampler
[ 72%] Building CXX object tests/CMakeFiles/test-state-restore-fragmented.dir/get-model.cpp.o
[ 72%] Linking CXX executable ../bin/test-state-restore-fragmented
[ 72%] Built target test-backend-sampler
[ 73%] Building CXX object tests/CMakeFiles/test-rope.dir/test-rope.cpp.o
[ 74%] Building CXX object tests/CMakeFiles/test-barrier.dir/get-model.cpp.o
[ 74%] Built target test-state-restore-fragmented
[ 74%] Linking CXX executable ../bin/test-barrier
[ 74%] Building C object tests/CMakeFiles/test-mtmd-c-api.dir/test-mtmd-c-api.c.o
[ 74%] Built target test-barrier
[ 74%] Building CXX object tests/CMakeFiles/test-quantize-fns.dir/get-model.cpp.o
[ 74%] Building CXX object tests/CMakeFiles/gguf-model-data.dir/gguf-model-data.cpp.o
[ 74%] Linking CXX executable ../bin/test-quantize-fns
[ 75%] Building CXX object tests/CMakeFiles/test-mtmd-c-api.dir/get-model.cpp.o
[ 75%] Built target test-quantize-fns
[ 75%] Linking CXX executable ../bin/test-mtmd-c-api
[ 75%] Building CXX object tests/CMakeFiles/test-alloc.dir/test-alloc.cpp.o
[ 75%] Built target test-mtmd-c-api
[ 75%] Building CXX object examples/batched/CMakeFiles/llama-batched.dir/batched.cpp.o
[ 76%] Building CXX object tests/CMakeFiles/test-quantize-perf.dir/get-model.cpp.o
[ 76%] Linking CXX executable ../bin/test-quantize-perf
[ 76%] Built target test-quantize-perf
[ 76%] Building CXX object examples/debug/CMakeFiles/llama-debug.dir/debug.cpp.o
[ 76%] Building CXX object tests/CMakeFiles/test-rope.dir/get-model.cpp.o
[ 76%] Linking CXX executable ../bin/test-rope
[ 76%] Built target test-rope
[ 76%] Building CXX object examples/embedding/CMakeFiles/llama-embedding.dir/embedding.cpp.o
[ 76%] Building CXX object tests/CMakeFiles/test-alloc.dir/get-model.cpp.o
[ 77%] Linking CXX executable ../bin/test-alloc
[ 77%] Linking CXX executable ../../bin/llama-batched
[ 77%] Built target test-alloc
[ 77%] Building CXX object examples/eval-callback/CMakeFiles/llama-eval-callback.dir/eval-callback.cpp.o
[ 77%] Built target llama-batched
[ 78%] Building CXX object examples/idle/CMakeFiles/llama-idle.dir/idle.cpp.o
[ 78%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-json-parser.cpp.o
[ 79%] Linking CXX executable ../../bin/llama-embedding
[ 79%] Built target llama-embedding
[ 79%] Building CXX object examples/lookahead/CMakeFiles/llama-lookahead.dir/lookahead.cpp.o
[ 79%] Linking CXX executable ../../bin/llama-idle
[ 79%] Linking CXX executable ../../bin/llama-eval-callback
[ 79%] Built target llama-idle
[ 79%] Linking CXX executable ../../bin/llama-debug
[ 79%] Building CXX object examples/lookup/CMakeFiles/llama-lookup.dir/lookup.cpp.o
[ 79%] Built target llama-eval-callback
[ 80%] Building CXX object examples/lookup/CMakeFiles/llama-lookup-create.dir/lookup-create.cpp.o
[ 80%] Built target llama-debug
[ 80%] Building CXX object examples/lookup/CMakeFiles/llama-lookup-merge.dir/lookup-merge.cpp.o
[ 80%] Linking CXX executable ../../bin/llama-lookahead
[ 80%] Building CXX object tests/CMakeFiles/test-backend-ops.dir/get-model.cpp.o
[ 80%] Linking CXX static library libgguf-model-data.a
[ 81%] Building CXX object examples/lookup/CMakeFiles/llama-lookup-stats.dir/lookup-stats.cpp.o
[ 81%] Built target llama-lookahead
[ 81%] Built target gguf-model-data
[ 81%] Building CXX object examples/parallel/CMakeFiles/llama-parallel.dir/parallel.cpp.o
[ 82%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-json-serialization.cpp.o
[ 82%] Linking CXX executable ../../bin/llama-lookup-merge
[ 82%] Linking CXX executable ../../bin/llama-lookup-create
[ 82%] Built target llama-lookup-merge
[ 82%] Linking CXX executable ../../bin/llama-lookup
[ 83%] Building CXX object examples/passkey/CMakeFiles/llama-passkey.dir/passkey.cpp.o
[ 83%] Built target llama-lookup-create
[ 83%] Building CXX object examples/retrieval/CMakeFiles/llama-retrieval.dir/retrieval.cpp.o
[ 83%] Built target llama-lookup
[ 84%] Building CXX object examples/save-load-state/CMakeFiles/llama-save-load-state.dir/save-load-state.cpp.o
[ 85%] Linking CXX executable ../bin/test-backend-ops
[ 85%] Linking CXX executable ../../bin/llama-lookup-stats
[ 85%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-python-dict-parser.cpp.o
[ 85%] Built target test-backend-ops
[ 86%] Building CXX object examples/speculative/CMakeFiles/llama-speculative.dir/speculative.cpp.o
[ 86%] Linking CXX executable ../../bin/llama-parallel
[ 86%] Built target llama-lookup-stats
[ 86%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-unicode.cpp.o
[ 86%] Built target llama-parallel
[ 86%] Building CXX object examples/speculative-simple/CMakeFiles/llama-speculative-simple.dir/speculative-simple.cpp.o
[ 86%] Linking CXX executable ../../bin/llama-passkey
[ 86%] Linking CXX executable ../../bin/llama-save-load-state
[ 86%] Built target llama-passkey
[ 86%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/get-model.cpp.o
[ 86%] Built target llama-save-load-state
[ 86%] Building CXX object examples/gen-docs/CMakeFiles/llama-gen-docs.dir/gen-docs.cpp.o
[ 87%] Building CXX object examples/training/CMakeFiles/llama-finetune.dir/finetune.cpp.o
[ 87%] Linking CXX executable ../../bin/llama-retrieval
[ 87%] Built target llama-retrieval
[ 87%] Building CXX object examples/diffusion/CMakeFiles/llama-diffusion-cli.dir/diffusion-cli.cpp.o
[ 87%] Linking CXX executable ../../bin/llama-speculative-simple
[ 87%] Linking CXX executable ../../bin/llama-finetune
[ 87%] Built target llama-speculative-simple
[ 87%] Building CXX object examples/convert-llama2c-to-ggml/CMakeFiles/llama-convert-llama2c-to-ggml.dir/convert-llama2c-to-ggml.cpp.o
[ 87%] Linking CXX executable ../../bin/llama-speculative
[ 87%] Built target llama-finetune
[ 88%] Building CXX object pocs/vdot/CMakeFiles/llama-vdot.dir/vdot.cpp.o
[ 88%] Linking CXX executable ../../bin/llama-gen-docs
[ 88%] Built target llama-speculative
[ 88%] Building CXX object pocs/vdot/CMakeFiles/llama-q8dot.dir/q8dot.cpp.o
[ 88%] Built target llama-gen-docs
[ 89%] Building CXX object tools/batched-bench/CMakeFiles/llama-batched-bench.dir/batched-bench.cpp.o
[ 89%] Building CXX object tools/gguf-split/CMakeFiles/llama-gguf-split.dir/gguf-split.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-q8dot
[ 90%] Linking CXX executable ../../bin/llama-vdot
[ 90%] Built target llama-q8dot
[ 90%] Built target llama-vdot
[ 90%] Building CXX object tools/imatrix/CMakeFiles/llama-imatrix.dir/imatrix.cpp.o
[ 90%] Building CXX object tools/llama-bench/CMakeFiles/llama-bench.dir/llama-bench.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-convert-llama2c-to-ggml
[ 90%] Linking CXX executable ../../bin/llama-batched-bench
[ 90%] Built target llama-convert-llama2c-to-ggml
[ 90%] Linking CXX executable ../../bin/llama-gguf-split
[ 90%] Building CXX object tools/completion/CMakeFiles/llama-completion.dir/completion.cpp.o
[ 90%] Built target llama-batched-bench
[ 90%] Building CXX object tools/perplexity/CMakeFiles/llama-perplexity.dir/perplexity.cpp.o
[ 90%] Built target llama-gguf-split
[ 90%] Building CXX object tools/quantize/CMakeFiles/llama-quantize.dir/quantize.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-diffusion-cli
[ 90%] Built target llama-diffusion-cli
[ 91%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-task.cpp.o
[ 91%] Building CXX object tools/tokenize/CMakeFiles/llama-tokenize.dir/tokenize.cpp.o
[ 92%] Linking CXX executable ../bin/test-peg-parser
[ 92%] Built target test-peg-parser
[ 93%] Building CXX object tools/parser/CMakeFiles/llama-debug-template-parser.dir/debug-template-parser.cpp.o
[ 93%] Linking CXX executable ../../bin/llama-tokenize
[ 93%] Built target llama-tokenize
[ 93%] Building CXX object tools/parser/CMakeFiles/llama-template-analysis.dir/template-analysis.cpp.o
[ 93%] Linking CXX executable ../../bin/llama-quantize
[ 93%] Built target llama-quantize
[ 93%] Building CXX object tools/tts/CMakeFiles/llama-tts.dir/tts.cpp.o
[ 93%] Linking CXX executable ../../bin/llama-completion
[ 93%] Linking CXX executable ../../bin/llama-perplexity
[ 93%] Built target llama-completion
[ 93%] Built target llama-perplexity
[ 93%] Building CXX object tools/mtmd/CMakeFiles/llama-mtmd-cli.dir/mtmd-cli.cpp.o
[ 93%] Building CXX object tools/mtmd/CMakeFiles/llama-mtmd-debug.dir/debug/mtmd-debug.cpp.o
[ 93%] Linking CXX executable ../../bin/llama-imatrix
[ 93%] Built target llama-imatrix
[ 93%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-queue.cpp.o
[ 93%] Linking CXX executable ../../bin/llama-bench
[ 93%] Built target llama-bench
[ 94%] Building CXX object tools/cvector-generator/CMakeFiles/llama-cvector-generator.dir/cvector-generator.cpp.o
[ 94%] Linking CXX executable ../../bin/llama-mtmd-debug
[ 94%] Built target llama-mtmd-debug
[ 94%] Building CXX object tools/export-lora/CMakeFiles/llama-export-lora.dir/export-lora.cpp.o
[ 94%] Linking CXX executable ../../bin/llama-debug-template-parser
[ 94%] Built target llama-debug-template-parser
[ 94%] Building CXX object tools/fit-params/CMakeFiles/llama-fit-params.dir/fit-params.cpp.o
[ 95%] Linking CXX executable ../../bin/llama-template-analysis
[ 95%] Built target llama-template-analysis
[ 95%] Building CXX object tools/results/CMakeFiles/llama-results.dir/results.cpp.o
[ 96%] Linking CXX executable ../../bin/llama-mtmd-cli
[ 96%] Built target llama-mtmd-cli
[ 96%] Building CXX object tests/CMakeFiles/test-gguf-model-data.dir/test-gguf-model-data.cpp.o
[ 96%] Linking CXX executable ../../bin/llama-cvector-generator
[ 96%] Built target llama-cvector-generator
[ 96%] Linking CXX executable ../../bin/llama-fit-params
[ 96%] Building CXX object tests/CMakeFiles/export-graph-ops.dir/export-graph-ops.cpp.o
[ 96%] Linking CXX executable ../../bin/llama-export-lora
[ 96%] Built target llama-fit-params
[ 96%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-common.cpp.o
[ 96%] Built target llama-export-lora
[ 96%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-context.cpp.o
[ 96%] Linking CXX executable ../../bin/llama-tts
[ 96%] Built target llama-tts
[ 96%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-tools.cpp.o
[ 96%] Linking CXX executable ../bin/test-gguf-model-data
[ 96%] Linking CXX executable ../../bin/llama-results
[ 96%] Built target test-gguf-model-data
[ 96%] Built target llama-results
[ 97%] Linking CXX executable ../bin/export-graph-ops
ld: warning: ignoring duplicate libraries: '../common/libcommon.a'
[ 97%] Built target export-graph-ops
[ 98%] Linking CXX static library libserver-context.a
[ 98%] Built target server-context
[ 99%] Generating bundle.css.hpp
[ 99%] Generating bundle.js.hpp
[ 99%] Generating index.html.hpp
[ 99%] Generating loading.html.hpp
[ 99%] Building CXX object tools/cli/CMakeFiles/llama-cli.dir/cli.cpp.o
[100%] Linking CXX executable ../../bin/llama-cli
[100%] Built target llama-cli
[100%] Building CXX object tools/server/CMakeFiles/llama-server.dir/server-models.cpp.o
[100%] Building CXX object tools/server/CMakeFiles/llama-server.dir/server.cpp.o
[100%] Building CXX object tools/server/CMakeFiles/llama-server.dir/server-http.cpp.o
[100%] Linking CXX executable ../../bin/llama-server
[100%] Built target llama-serverllama.cpp built successfully at ~/src/llama.cpp/build/bin/llama-server. Now I’m launching it against the Gemma GGUF and probing /v1/models.
>~/src/llama.cpp/build/bin/llama-server --help | rg -n "--alias|--host|--port|-hf|hugging|repo|jinja|ctx-size|parallel|n-gpu|ngl|model" -Srg: unrecognized flag --alias|--host|--port|-hf|hugging|repo|jinja|ctx-size|parallel|n-gpu|ngl|model
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.509 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
Command exited with code 2Adjusting command usage
I realized that the "rg" pattern requires either the -e flag or quotes with --. I see that maybe there's a way to simplify using grep, but I must use "rg" instead of grep. I’ll need to work with bash and consider redirecting output to a file. I can also pipe to "rg" with a command like 'alias|host' to get the results I need. It feels like I'm getting closer to figuring this out!
>~/src/llama.cpp/build/bin/llama-server --help 2>&1 | rg -n 'alias|host|port|hf|hugging|ctx-size|parallel|ngl|model'22:-cl, --cache-list show list of models in cache
42:-c, --ctx-size N size of the prompt context (default: 0, 0 = loaded from model)
64: the model
69: model)
73:--yarn-orig-ctx N YaRN: original context size of model (default: 0 = model training
90:--no-host bypass host buffer allowing extra buffers to be used
102:--mlock force system to keep model in RAM rather than swapping or compressing
104:--mmap, --no-mmap whether to memory-map model. (if mmap disabled, slower load but may
132:-ngl, --gpu-layers, --n-gpu-layers N max. number of layers to store in VRAM, either an exact number,
135:-sm, --split-mode {none,layer,row} how to split the model across multiple GPUs, one of:
140:-ts, --tensor-split N0,N1,N2,... fraction of the model to offload to each GPU, comma-separated list of
141: proportions, e.g. 3,1
143:-mg, --main-gpu INDEX the GPU to use for the model (with split-mode = none), or for
151: single value is broadcast across all devices, default: 1024
155:--check-tensors check model tensor data for invalid values (default: false)
156:--override-kv KEY=TYPE:VALUE,... advanced option to override model metadata by key. to specify multiple
160:--op-offload, --no-op-offload whether to offload host tensor operations to device (default: true)
173:-m, --model FNAME model path to load
175:-mu, --model-url MODEL_URL model download url (default: unused)
177:-dr, --docker-repo [<repo>/]<model>[:quant]
178: Docker Hub model repository. repo is optional, default to ai/. quant
183:-hf, -hfr, --hf-repo <user>/<model>[:quant]
184: Hugging Face model repository; quant is optional, case-insensitive,
192:-hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant]
193: Same as --hf-repo, but for the draft model (default: unused)
195:-hff, --hf-file FILE Hugging Face model file. If specified, it will override the quant in
196: --hf-repo (default: unused)
198:-hfv, -hfrv, --hf-repo-v <user>/<model>[:quant]
199: Hugging Face model repository for the vocoder model (default: unused)
201:-hffv, --hf-file-v FILE Hugging Face model file for the vocoder model (default: unused)
203:-hft, --hf-token TOKEN Hugging Face access token (default: value from HF_TOKEN environment
230:-ctkd, --cache-type-k-draft TYPE KV cache data type for K for the draft model
234:-ctvd, --cache-type-v-draft TYPE KV cache data type for V for the draft model
323: use single unified KV buffer shared across all sequences (default:
333: Prefix/Suffix/Middle) as some models prefer this. (default: disabled)
334:--pooling {none,mean,cls,last,rank} pooling type for embeddings, use model default if unspecified
336:-np, --parallel N number of server slots (default: -1, -1 = auto)
343: note: if -hf is used, this argument can be omitted
349: using -hf (default: enabled)
355: models with dynamic resolution (default: read from model)
358: models with dynamic resolution (default: read from model)
361: override tensor buffer type for draft model
363: model
366: CPU for the draft model
368:-a, --alias STRING set model name aliases, comma-separated (to be used by API)
370:--tags STRING set model tags, comma-separated (informational, not used for routing)
372:--host HOST ip address to listen, or bind to an UNIX socket if the address ends
375:--port PORT port to listen (default: 8080)
377:--reuse-port allow multiple sockets to bind to the same port (default: disabled)
401:--embedding, --embeddings restrict to only support embedding use case; use only with dedicated
402: embedding models (default: disabled)
436:--models-dir PATH directory containing models for the router server (default: disabled)
438:--models-preset PATH path to INI file containing model presets for the router server
441:--models-max N for router server, maximum number of models to load simultaneously
444:--models-autoload, --no-models-autoload
445: for router server, whether to automatically load models (default:
467:--chat-template JINJA_TEMPLATE set custom jinja chat template (default: template taken from model's
485: model's metadata)
502: model will output everything in the content section, including any
530:-cd, --ctx-size-draft N size of the prompt context for the draft model (default: 0, 0 = loaded
531: from model)
533:-devd, --device-draft <dev1,dev2,..> comma-separated list of devices to use for offloading the draft model
536:-ngld, --gpu-layers-draft, --n-gpu-layers-draft N
537: max. number of draft model layers to store in VRAM, either an exact
540:-md, --model-draft FNAME draft model for speculative decoding (default: unused)
542:--spec-replace TARGET DRAFT translate the string in TARGET into DRAFT if the draft model and main
543: model are not compatible
545: type of speculative decoding to use when no draft model is provided
554:-mv, --model-vocoder FNAME vocoder model for audio generation (default: unused)
556:--embd-gemma-default use default EmbeddingGemma model (note: can download weights from thePlanning a server run
I’m considering running a server, potentially using nohup to keep it in the background, especially since it might involve a huge download. It’s a good idea to set a timeout of about 20 minutes and poll the progress while it runs. I might need to check the cache location too. Using default context size seems best, but I’ll set an alias for convenience. There’s a chance it might fail due to memory issues, so I’m staying cautious!
>mkdir -p /tmp/llama-pi && cd ~/src/llama.cpp && rm -f /tmp/llama-pi/gemma-server.log /tmp/llama-pi/gemma-server.pid && nohup ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080 > /tmp/llama-pi/gemma-server.log 2>&1 & echo $! > /tmp/llama-pi/gemma-server.pid && echo "pid=$(cat /tmp/llama-pi/gemma-server.pid)" && sleep 5 && lsof -nP -iTCP:8080 -sTCP:LISTEN || true && echo '--- LOG TAIL ---' && tail -n 80 /tmp/llama-pi/gemma-server.log/bin/bash: /tmp/llama-pi/gemma-server.pid: No such file or directory
--- LOG TAIL ---
tail: /tmp/llama-pi/gemma-server.log: No such file or directory
Command exited with code 1Considering shell commands
I'm thinking about a possible issue with grouping commands like mkdir -p and rm. It seems like the redirection happens before mkdir gets executed in a subshell, which could lead to a missing path. I wonder if separating them with a semicolon would be safer, and maybe referencing $HOME might help too. It's essential to clarify and ensure that the commands work as intended! I want to get this right!
>mkdir -p /tmp/llama-pi; cd ~/src/llama.cpp; nohup ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080 >/tmp/llama-pi/gemma-server.log 2>&1 & echo $! >/tmp/llama-pi/gemma-server.pid; echo "pid=$(cat /tmp/llama-pi/gemma-server.pid)"; sleep 5; lsof -nP -iTCP:8080 -sTCP:LISTEN || true; echo '--- LOG TAIL ---'; tail -n 80 /tmp/llama-pi/gemma-server.logpid=31038
--- LOG TAIL ---
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library>for i in $(seq 1 24); do echo "== poll $i =="; if lsof -nP -iTCP:8080 -sTCP:LISTEN >/tmp/llama-pi/listen.txt 2>/dev/null; then cat /tmp/llama-pi/listen.txt; break; fi; sleep 10; tail -n 40 /tmp/llama-pi/gemma-server.log; echo; done; echo '--- FINAL LOG ---'; tail -n 120 /tmp/llama-pi/gemma-server.log== poll 1 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 2 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 3 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 4 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 5 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 6 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 7 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 8 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 9 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 10 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 11 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 12 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 13 ==
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
== poll 14 ==
Command abortedhow we going?
Inspecting server status
I need to check the server's status because the build seems okay, but it’s either stuck downloading or there may have been a failure with the preset. I should investigate the running process to see if it’s actually listening. There might not be any changes right now, so it could be helpful to use Bash to perform a deeper inspection and figure out what’s going on. Let's see what I can find!
Build is done. The Gemma test is not up yet.
Current state:
llama.cpp built successfullyllama-server binary exists at ~/src/llama.cpp/build/bin/llama-server-hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M is not reaching port 8080common_download_file_single_online: HEAD failed, status: 404no remote preset found, skippingThat preset line is not fatal by itself. The likely issue is the Hugging Face model resolution or download step is hanging before the server binds.
I can continue by checking:
llama-server wants --hf-file for this repo layout-m <local-file>That is the route I’d take next.
>ps -p $(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null) -o pid,etime,%cpu,%mem,command || true; echo '---'; tail -n 120 /tmp/llama-pi/gemma-server.log 2>/dev/null || true; echo '---'; curl -sf http://127.0.0.1:8080/v1/models || truePID ELAPSED %CPU %MEM COMMAND
31038 02:39 9.9 0.1 ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080
---
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
---Considering user request
I’m thinking about how to respond to the user. They asked for a status update, which means I should keep it concise and focused. I could consider using web tools, but it seems like they want the status rather than detailed information. It might be worth noting that the process is still running but likely blocked. I want to ensure I'm addressing their request effectively without overcomplicating things.
Build succeeded.
Blocked part:
llama-server is running as PID 31038127.0.0.1:8080 yetcommon_download_file_single_online: HEAD failed, status: 404no remote preset found, skippingSo the issue is not the compile. It is model fetch/startup for ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M.
Next step is to stop relying on auto-resolution and verify the exact HF artifact, then either:
-hf ... --hf-file ..., or-m /path/to/model.ggufI can do that next and get it to a working /v1/models response.
i thin 8080 is assigned to a different process? check
Inspecting port usage
I need to inspect what's happening on port 8080. My best plan is to use the lsof command, which can help me see which processes are using that port. It will provide a list of all the open files and their associated processes, giving me clarity on who’s listening on that port or if there’s any related activity. It's a handy tool for troubleshooting and keeping things running smoothly!
Checking what owns port 8080, then I’ll either kill the stale llama-server or move to a free port.
>echo '--- listeners on 8080 ---'; lsof -nP -iTCP:8080 -sTCP:LISTEN || true; echo '--- all 8080 fds ---'; lsof -nP -iTCP:8080 || true; echo '--- llama-server pid ---'; ps -p $(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null) -o pid,ppid,etime,%cpu,%mem,command || true--- listeners on 8080 ---
--- all 8080 fds ---
--- llama-server pid ---
PID PPID ELAPSED %CPU %MEM COMMAND
31038 1 03:18 4.0 0.1 ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080Port 8080 is free.
Results:
TCP:8080llama-server process is still alive, but it never bound the portSo the problem is upstream of socket bind, likely model resolution/download/startup, not port contention.
is llama-server process still running and downloading?
I’m checking whether the llama-server process is still alive and whether it shows active download or file write activity.
>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
if [ -z "$PID" ]; then echo 'no pid file'; exit 0; fi
echo '--- process ---'
ps -p "$PID" -o pid,ppid,state,etime,%cpu,%mem,command || true
echo '--- open files mentioning cache/huggingface/gguf ---'
lsof -p "$PID" 2>/dev/null | rg -i 'hugging|cache|gguf|tmp|download' || true
echo '--- network sockets ---'
lsof -nP -p "$PID" -i || true
echo '--- child processes ---'
pgrep -P "$PID" -fl . || true
echo '--- likely cache dirs ---'
for d in "$HOME/.cache/llama.cpp" "$HOME/.cache/huggingface" "$HOME/Library/Caches/llama.cpp" "$HOME/Library/Caches/ggml"; do
if [ -e "$d" ]; then
echo "# $d"
find "$d" -type f -mmin -10 2>/dev/null | head -n 20
fi
done
echo '--- recent log ---'
tail -n 80 /tmp/llama-pi/gemma-server.log 2>/dev/null || true--- process ---
PID PPID STAT ELAPSED %CPU %MEM COMMAND
31038 1 R 03:48 13.2 0.1 ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080
--- open files mentioning cache/huggingface/gguf ---
llama-ser 31038 badlogic txt REG 1,15 73596 161859496 /Library/Preferences/Logging/.plist-cache.q2HJ6EY6
llama-ser 31038 badlogic 1w REG 1,15 1403 162082292 /private/tmp/llama-pi/gemma-server.log
llama-ser 31038 badlogic 2w REG 1,15 1403 162082292 /private/tmp/llama-pi/gemma-server.log
llama-ser 31038 badlogic 9w REG 1,15 5879836672 162082302 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
--- network sockets ---
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rapportd 624 badlogic 11u IPv4 0x984a55d27d4d493a 0t0 TCP *:62077 (LISTEN)
rapportd 624 badlogic 15u IPv6 0x8e18a78e12829c41 0t0 TCP *:62077 (LISTEN)
ControlCe 628 badlogic 8u IPv4 0xdff51c11926118d0 0t0 TCP *:7000 (LISTEN)
ControlCe 628 badlogic 9u IPv6 0xd464042e331234dc 0t0 TCP *:7000 (LISTEN)
ControlCe 628 badlogic 10u IPv4 0x7719fa6e128bf553 0t0 TCP *:5000 (LISTEN)
ControlCe 628 badlogic 11u IPv6 0xe46e3f68fb926d78 0t0 TCP *:5000 (LISTEN)
identitys 681 badlogic 11u IPv4 0x3a36cd227866bd11 0t0 UDP *:*
identitys 681 badlogic 15u IPv4 0x7e9aae31cde278f4 0t0 UDP *:*
identitys 681 badlogic 20u IPv4 0xa67cceac4cc98b99 0t0 UDP *:*
sharingd 686 badlogic 4u IPv4 0x2509046dbb322434 0t0 UDP *:*
sharingd 686 badlogic 10u IPv6 0x979fc4721cfab63d 0t0 UDP *:63633
replicato 736 badlogic 8u IPv6 0x868de0b16197e874 0t0 UDP *:56203
LM\x20Stu 849 badlogic 75u IPv4 0x9be759180cc19014 0t0 TCP 127.0.0.1:41343 (LISTEN)
LM\x20Stu 849 badlogic 77u IPv4 0x443d566eeec000f 0t0 TCP 127.0.0.1:1234 (LISTEN)
remoting_ 990 badlogic 31u IPv4 0x5964a2b7c8d97bf4 0t0 UDP 192.168.1.32:50154->216.239.34.223:443
Google 12429 badlogic 22u IPv4 0xa0e3d90e937932a5 0t0 TCP 192.168.1.32:62228->172.66.0.227:443 (ESTABLISHED)
Google 12429 badlogic 26u IPv4 0x9b00c0f47b534e86 0t0 TCP 192.168.1.32:62087->162.159.135.234:443 (ESTABLISHED)
Google 12429 badlogic 27u IPv4 0xc5d306ac8d4f7de7 0t0 TCP 192.168.1.32:62075->35.186.224.39:443 (ESTABLISHED)
Google 12429 badlogic 28u IPv4 0x347e48454af04e81 0t0 TCP 192.168.1.32:62089->173.194.76.188:5228 (ESTABLISHED)
Google 12429 badlogic 34u IPv4 0x2e2f99f4ceeb3261 0t0 UDP *:5353
Google 12429 badlogic 42u IPv6 0x7dd9b275a33f6ca 0t0 UDP *:5353
Google 12429 badlogic 47u IPv6 0xa513963b0294e424 0t0 UDP *:5353
Google 12429 badlogic 48u IPv6 0xf9cb8852e39bc614 0t0 UDP *:5353
Google 12429 badlogic 49u IPv6 0xc77bcb45ef88f9c5 0t0 UDP *:5353
Google 12429 badlogic 50u IPv6 0x79f38efcfaff8969 0t0 UDP *:5353
Google 12429 badlogic 52u IPv6 0x896ea3a75f2e4e58 0t0 UDP *:5353
Google 12429 badlogic 53u IPv6 0x9e512b4469f348cd 0t0 UDP *:5353
node 20530 badlogic 12u IPv6 0x94c2e0f237a4cad8 0t0 TCP *:4444 (LISTEN)
node 21337 badlogic 30u IPv6 0xe7d3e11015195478 0t0 TCP *:3210 (LISTEN)
llama-ser 31035 badlogic 10u IPv4 0x6ee76a9e45ac1c79 0t0 TCP 192.168.1.32:62420->99.84.91.56:443 (ESTABLISHED)
llama-ser 31038 badlogic cwd DIR 1,15 1952 162065701 /Users/badlogic/src/llama.cpp
llama-ser 31038 badlogic txt REG 1,15 10988976 162082134 /Users/badlogic/src/llama.cpp/build/bin/llama-server
llama-ser 31038 badlogic txt REG 1,15 59840 162072022 /Users/badlogic/src/llama.cpp/build/bin/libggml.0.9.11.dylib
llama-ser 31038 badlogic txt REG 1,15 58664 162071667 /Users/badlogic/src/llama.cpp/build/bin/libggml-blas.0.9.11.dylib
llama-ser 31038 badlogic txt REG 1,15 73596 161859496 /Library/Preferences/Logging/.plist-cache.q2HJ6EY6
llama-ser 31038 badlogic txt REG 1,15 110 1152921500312100187 /System/Library/CoreServices/SystemVersion.bundle/en_GB.lproj/SystemVersion.strings
llama-ser 31038 badlogic txt REG 1,15 2299152 1152921500312560823 /usr/lib/dyld
llama-ser 31038 badlogic txt REG 1,15 900224 124108504 /opt/homebrew/Cellar/openssl@3/3.6.0/lib/libssl.3.dylib
llama-ser 31038 badlogic txt REG 1,15 842592 162076421 /Users/badlogic/src/llama.cpp/build/bin/libmtmd.0.0.8639.dylib
llama-ser 31038 badlogic txt REG 1,15 652552 162071385 /Users/badlogic/src/llama.cpp/build/bin/libggml-base.0.9.11.dylib
llama-ser 31038 badlogic txt REG 1,15 32768 161860530 /private/var/db/mds/messages/501/se_SecurityMessages
llama-ser 31038 badlogic txt REG 1,15 916848 162071971 /Users/badlogic/src/llama.cpp/build/bin/libggml-cpu.0.9.11.dylib
llama-ser 31038 badlogic txt REG 1,15 815240 162071922 /Users/badlogic/src/llama.cpp/build/bin/libggml-metal.0.9.11.dylib
llama-ser 31038 badlogic txt REG 1,15 165245 161981464 /private/var/db/analyticsd/events.allowlist
llama-ser 31038 badlogic txt REG 1,15 234944 150409245 /private/var/db/timezone/tz/2025c.1.0/icutz/icutz44l.dat
llama-ser 31038 badlogic txt REG 1,15 4843248 124108503 /opt/homebrew/Cellar/openssl@3/3.6.0/lib/libcrypto.3.dylib
llama-ser 31038 badlogic txt REG 1,15 2377424 162075522 /Users/badlogic/src/llama.cpp/build/bin/libllama.0.0.8639.dylib
llama-ser 31038 badlogic txt REG 1,15 1549096 1152921500312125732 /System/Library/Extensions/AGXMetalG13X.bundle/Contents/Resources/ds.g13s
llama-ser 31038 badlogic txt REG 1,15 21328240 1152921500312125719 /System/Library/Extensions/AGXMetalG13X.bundle/Contents/MacOS/AGXMetalG13X
llama-ser 31038 badlogic txt REG 1,15 34125552 1152921500312571275 /usr/share/icu/icudt76l.dat
llama-ser 31038 badlogic txt REG 1,15 45596672 95907974 /private/var/folders/49/l4171l3n219_xwq30fmyzr8w0000gn/C/com.apple.metal/32024/libraries1.data
llama-ser 31038 badlogic 0r CHR 3,2 0t0 336 /dev/null
llama-ser 31038 badlogic 1w REG 1,15 1403 162082292 /private/tmp/llama-pi/gemma-server.log
llama-ser 31038 badlogic 2w REG 1,15 1403 162082292 /private/tmp/llama-pi/gemma-server.log
llama-ser 31038 badlogic 3 NPOLICY
llama-ser 31038 badlogic 4u REG 1,15 45596672 95907974 /private/var/folders/49/l4171l3n219_xwq30fmyzr8w0000gn/C/com.apple.metal/32024/libraries1.data
llama-ser 31038 badlogic 5u REG 1,15 59180 162082307 /private/var/folders/49/l4171l3n219_xwq30fmyzr8w0000gn/C/com.apple.metal/32024/libraries1.list
llama-ser 31038 badlogic 6u systm 0x14d88ed6aa56afd9 0t0 [ctl com.apple.netsrc id 7 unit 88]
llama-ser 31038 badlogic 7u unix 0x40a00664c97276a 0t0 ->0x23489e44e6cc2563
llama-ser 31038 badlogic 9w REG 1,15 5894279168 162082302 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
llama-ser 31038 badlogic 11u IPv4 0x9199623e63357e40 0t0 TCP 192.168.1.32:62442->99.84.91.56:443 (ESTABLISHED)
--- child processes ---
--- likely cache dirs ---
# /Users/badlogic/.cache/huggingface
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/refs/main
# /Users/badlogic/Library/Caches/llama.cpp
--- recent log ---
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
common_download_file_single_online: unable to rename file: /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531.downloadInProgress to /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531Revising response clarity
I need to provide a concise answer about running and downloading, ensuring I mention the evidence. It’s also a good idea to rename any errors to avoid confusion.
Let’s see if I can streamline my message! I want it to be clear and informative while also being brief. I’ll focus on the essential points and make sure there’s no ambiguity in what I’m saying. Getting this right is important!
Yes.
Current state:
31038 is still running99.84.91.56:443/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
That file is already about 5.89 GB, so it is definitely in the download path.
There is also a real issue in the log:
unable to rename file: ...4107c1c3....downloadInProgress to ...4107c1c3...
So it looks like:
If you want, I can monitor until the GGUF finishes, then retry the server cleanly. Or I can stop it now, inspect the Hugging Face cache layout, and relaunch in a cleaner way using a pre-downloaded local model path.
update?
Checking current download size, process state, and whether the server bound yet.
>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
echo '--- process ---'
ps -p "$PID" -o pid,state,etime,%cpu,%mem,command || true
echo '--- 8080 ---'
lsof -nP -iTCP:8080 -sTCP:LISTEN || true
echo '--- active download files ---'
lsof -p "$PID" 2>/dev/null | rg 'downloadInProgress|models--ggml-org--gemma-4-26b-a4b-it-GGUF' || true
echo '--- cache file sizes ---'
find "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs" -maxdepth 1 -type f 2>/dev/null | xargs -I{} stat -f '%z %N' '{}' 2>/dev/null | sort -nr | head -n 10 || true
echo '--- log tail ---'
tail -n 60 /tmp/llama-pi/gemma-server.log 2>/dev/null || true--- process ---
PID STAT ELAPSED %CPU %MEM COMMAND
31038 S 06:12 6.3 0.1 ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080
--- 8080 ---
--- active download files ---
llama-ser 31038 badlogic 9w REG 1,15 11047583744 162082302 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
--- cache file sizes ---
11050909696 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
2082877568 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531
--- log tail ---
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
common_download_file_single_online: unable to rename file: /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531.downloadInProgress to /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531Current status:
llama-server is still running808011.05 GBRelevant files:
~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997....downloadInProgress~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3...2.08 GBLog has not advanced beyond the earlier cache rename warning, but the open file growth confirms the download is still moving.
So:
If you want, I can keep polling until the download completes, then verify /v1/models and run a test prompt.
update
Polling current state of the Gemma download and server bind.
>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
echo '--- process ---'
ps -p "$PID" -o pid,state,etime,%cpu,%mem,command || true
echo '--- 8080 ---'
lsof -nP -iTCP:8080 -sTCP:LISTEN || true
echo '--- download file ---'
lsof -p "$PID" 2>/dev/null | rg 'downloadInProgress|models--ggml-org--gemma-4-26b-a4b-it-GGUF' || true
echo '--- latest blob sizes ---'
find "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs" -maxdepth 1 -type f 2>/dev/null | xargs -I{} stat -f '%Sm %z %N' -t '%Y-%m-%d %H:%M:%S' '{}' 2>/dev/null | sort -r | head -n 12 || true
echo '--- log tail ---'
tail -n 80 /tmp/llama-pi/gemma-server.log 2>/dev/null || true--- process ---
PID STAT ELAPSED %CPU %MEM COMMAND
31038 S 13:36 14.0 0.1 ./build/bin/llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080
--- 8080 ---
--- download file ---
llama-ser 31038 badlogic 9w REG 1,15 25766227968 162082302 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
--- latest blob sizes ---
2026-04-02 21:49:21 25771622400 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress
2026-04-02 21:38:30 2082877568 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531
--- log tail ---
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
common_download_file_single_online: unable to rename file: /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531.downloadInProgress to /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531Update:
808025.77 GBSo it is making substantial progress. No new error beyond the earlier rename warning.
Status:
This model/quant is much larger than expected, so startup is gated on that full artifact download.
now?
Checking whether the download finished and whether llama-server finally bound the API port.
>PID=$(cat /tmp/llama-pi/gemma-server.pid 2>/dev/null || true)
echo '--- process ---'
ps -p "$PID" -o pid,state,etime,%cpu,%mem,command || true
echo '--- 8080 ---'
lsof -nP -iTCP:8080 -sTCP:LISTEN || true
echo '--- curl /v1/models ---'
curl -sS http://127.0.0.1:8080/v1/models || true
echo '
--- cache status ---'
find "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs" -maxdepth 1 -type f 2>/dev/null | xargs -I{} stat -f '%Sm %z %N' -t '%Y-%m-%d %H:%M:%S' '{}' 2>/dev/null | sort -r | head -n 12 || true
echo '--- open download files ---'
lsof -p "$PID" 2>/dev/null | rg 'downloadInProgress|models--ggml-org--gemma-4-26b-a4b-it-GGUF' || true
echo '--- log tail ---'
tail -n 120 /tmp/llama-pi/gemma-server.log 2>/dev/null || true--- process ---
PID STAT ELAPSED %CPU %MEM COMMAND
--- 8080 ---
--- curl /v1/models ---
curl: (7) Failed to connect to 127.0.0.1 port 8080 after 0 ms: Couldn't connect to server
--- cache status ---
2026-04-02 21:53:36 33348895168 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5
2026-04-02 21:38:30 2082877568 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531
--- open download files ---
--- log tail ---
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 9.612 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 55662.79 MB
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
common_download_file_single_online: unable to rename file: /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5.downloadInProgress to /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5
error: failed to download model from Hugging FaceAddressing download issues
I need to keep the answer concise since the user is asking now. I've finished the download, but the process exited due to a rename issue, which is strange because the file already exists. It seems necessary to start with the local blob path, and maybe using a symlink snapshot could help. I'll mention this as the next step for resolving the issue.
Now we have the failure point.
Status:
llama-server exited8080 is still freeerror: failed to download model from Hugging Face
The cache now contains:
~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c699...33.35 GB2.08 GBSo the model artifact is on disk, but llama-server tripped over its cache rename/finalization logic and aborted.
Best next move:
-hf ...llama-server -m <local-gguf> directlyThat should bypass the broken download-finalization path entirely. If you want, I’ll do that next and get the server actually responding.
try again
Considering cache operations
I need to take action based on what's been explicitly asked. I want to look into a cache snapshot and possibly use the hf cache layout. I also need to check if I can perform file operations using Bash to find the actual file. There's a chance the process might be dead, so I'll have to launch it with a local file. I'll make sure to mention everything concisely!
Retrying now using the already downloaded local cache file instead of -hf, then I’ll probe /v1/models and a test completion.
>set -e
cd ~/src/llama.cpp
CACHE_DIR="$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF"
echo '--- cache layout ---'
find "$CACHE_DIR" -maxdepth 3 \( -type f -o -type l \) | sort | head -n 200
echo '--- refs/main ---'
if [ -f "$CACHE_DIR/refs/main" ]; then cat "$CACHE_DIR/refs/main"; fi
echo '--- snapshots ---'
find "$CACHE_DIR/snapshots" -maxdepth 2 \( -type f -o -type l \) 2>/dev/null | sort || true
--- cache layout ---
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/refs/main
--- refs/main ---
4006d4d95819e0a4975eae220762f1552dc2f699--- snapshots ---Inspecting blob types
I need to check the blob types, maybe looking for strings. I’m wondering if one could be for “gguf” and the other for a tokenizer. To get a clearer understanding, I’ll use a file command. It's a straightforward approach, but I want to ensure I correctly identify each type. This way, I can manage the data appropriately, ensuring I’m not missing anything important. Let's get started on that!
>CACHE_DIR="$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs"
for f in "$CACHE_DIR"/*; do
echo "--- $f ---"
file "$f"
ls -lh "$f"
xxd -l 32 "$f" | head -n 2
echo
done--- /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 ---
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5: data
-rw-r--r--@ 1 badlogic staff 31G Apr 2 21:53 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5
00000000: 4747 5546 0300 0000 9202 0000 0000 0000 GGUF............
00000010: 2f00 0000 0000 0000 1400 0000 0000 0000 /...............
--- /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 ---
/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531: data
-rw-r--r--@ 1 badlogic staff 1.9G Apr 2 21:38 /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531
00000000: 4747 5546 0300 0000 6401 0000 0000 0000 GGUF....d.......
00000010: 1700 0000 0000 0000 1400 0000 0000 0000 ................Considering project specifications
I’m thinking I might need one mmproj. For the launch, it seems like the 31G main model could be enough—maybe even multimodal? I wonder if gemma-4 needs a projector or if it could just be text-only. I’m curious about "a4b-it" in relation to text. Should I consider splitting into two gguf? I should also inspect the metadata—maybe with llama-gguf, as it could provide the outputs with architecture names.
Found the cached GGUFs. One is a 31 GB main model, the other is a 1.9 GB auxiliary GGUF. I’m identifying which one is the actual text model, then I’ll start llama-server with the local file path.
>cd ~/src/llama.cpp
for f in "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs"/*; do
echo "=== $f ==="
./build/bin/llama-gguf "$f" | head -n 80
echo
done=== /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 ===
usage: ./build/bin/llama-gguf data.gguf r|w [n]
r: read data.gguf file
w: write data.gguf file
n: no check of tensor data
=== /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 ===
usage: ./build/bin/llama-gguf data.gguf r|w [n]
r: read data.gguf file
w: write data.gguf file
n: no check of tensor data>cd ~/src/llama.cpp
for f in "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs"/*; do
echo "=== $f ==="
./build/bin/llama-gguf "$f" r | head -n 120
echo
done=== /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 ===
gguf_ex_read_0: version: 3
gguf_ex_read_0: alignment: 32
gguf_ex_read_0: data offset: 15816992
gguf_ex_read_0: n_kv: 47
gguf_ex_read_0: kv[0]: key = general.architecture
gguf_ex_read_0: kv[1]: key = general.type
gguf_ex_read_0: kv[2]: key = general.sampling.top_k
gguf_ex_read_0: kv[3]: key = general.sampling.top_p
gguf_ex_read_0: kv[4]: key = general.sampling.temp
gguf_ex_read_0: kv[5]: key = general.size_label
gguf_ex_read_0: kv[6]: key = general.license
gguf_ex_read_0: kv[7]: key = general.license.link
gguf_ex_read_0: kv[8]: key = general.tags
gguf_ex_read_0: kv[9]: key = gemma4.block_count
gguf_ex_read_0: kv[10]: key = gemma4.context_length
gguf_ex_read_0: kv[11]: key = gemma4.embedding_length
gguf_ex_read_0: kv[12]: key = gemma4.feed_forward_length
gguf_ex_read_0: kv[13]: key = gemma4.attention.head_count
gguf_ex_read_0: kv[14]: key = gemma4.attention.head_count_kv
gguf_ex_read_0: kv[15]: key = gemma4.rope.freq_base
gguf_ex_read_0: kv[16]: key = gemma4.rope.freq_base_swa
gguf_ex_read_0: kv[17]: key = gemma4.attention.layer_norm_rms_epsilon
gguf_ex_read_0: kv[18]: key = gemma4.expert_count
gguf_ex_read_0: kv[19]: key = gemma4.expert_used_count
gguf_ex_read_0: kv[20]: key = gemma4.attention.key_length
gguf_ex_read_0: kv[21]: key = gemma4.attention.value_length
gguf_ex_read_0: kv[22]: key = gemma4.final_logit_softcapping
gguf_ex_read_0: kv[23]: key = gemma4.attention.sliding_window
gguf_ex_read_0: kv[24]: key = gemma4.attention.shared_kv_layers
gguf_ex_read_0: kv[25]: key = gemma4.embedding_length_per_layer_input
gguf_ex_read_0: kv[26]: key = gemma4.attention.sliding_window_pattern
gguf_ex_read_0: kv[27]: key = gemma4.attention.key_length_swa
gguf_ex_read_0: kv[28]: key = gemma4.attention.value_length_swa
gguf_ex_read_0: kv[29]: key = gemma4.expert_feed_forward_length
gguf_ex_read_0: kv[30]: key = gemma4.rope.dimension_count
gguf_ex_read_0: kv[31]: key = gemma4.rope.dimension_count_swa
gguf_ex_read_0: kv[32]: key = tokenizer.ggml.model
gguf_ex_read_0: kv[33]: key = tokenizer.ggml.tokens
gguf_ex_read_0: kv[34]: key = tokenizer.ggml.scores
gguf_ex_read_0: kv[35]: key = tokenizer.ggml.token_type
gguf_ex_read_0: kv[36]: key = tokenizer.ggml.merges
gguf_ex_read_0: kv[37]: key = tokenizer.ggml.bos_token_id
gguf_ex_read_0: kv[38]: key = tokenizer.ggml.eos_token_id
gguf_ex_read_0: kv[39]: key = tokenizer.ggml.unknown_token_id
gguf_ex_read_0: kv[40]: key = tokenizer.ggml.padding_token_id
gguf_ex_read_0: kv[41]: key = tokenizer.ggml.mask_token_id
gguf_ex_read_0: kv[42]: key = tokenizer.chat_template
gguf_ex_read_0: kv[43]: key = tokenizer.ggml.add_space_prefix
gguf_ex_read_0: kv[44]: key = tokenizer.ggml.add_bos_token
gguf_ex_read_0: kv[45]: key = general.quantization_version
gguf_ex_read_0: kv[46]: key = general.file_type
gguf_ex_read_0: find key: some.parameter.string not found.
gguf_ex_read_0: n_tensors: 658
gguf_ex_read_0: tensor[0]: name = output_norm.weight, size = 11264, offset = 0
gguf_ex_read_0: tensor[1]: name = rope_freqs.weight, size = 1024, offset = 11264
gguf_ex_read_0: tensor[2]: name = token_embd.weight, size = 605552640, offset = 12288
gguf_ex_read_0: tensor[3]: name = blk.0.attn_k.weight, size = 3244032, offset = 605564928
gguf_ex_read_0: tensor[4]: name = blk.0.attn_k_norm.weight, size = 1024, offset = 608808960
gguf_ex_read_0: tensor[5]: name = blk.0.attn_norm.weight, size = 11264, offset = 608809984
gguf_ex_read_0: tensor[6]: name = blk.0.attn_output.weight, size = 6488064, offset = 608821248
gguf_ex_read_0: tensor[7]: name = blk.0.attn_q.weight, size = 6488064, offset = 615309312
gguf_ex_read_0: tensor[8]: name = blk.0.attn_q_norm.weight, size = 1024, offset = 621797376
gguf_ex_read_0: tensor[9]: name = blk.0.attn_v.weight, size = 4730880, offset = 621798400
gguf_ex_read_0: tensor[10]: name = blk.0.ffn_down.weight, size = 6319104, offset = 626529280
gguf_ex_read_0: tensor[11]: name = blk.0.ffn_down_exps.scale, size = 512, offset = 632848384
gguf_ex_read_0: tensor[12]: name = blk.0.ffn_down_exps.weight, size = 269615104, offset = 632848896
gguf_ex_read_0: tensor[13]: name = blk.0.ffn_gate.weight, size = 3345408, offset = 902464000
gguf_ex_read_0: tensor[14]: name = blk.0.ffn_gate_inp.scale, size = 11264, offset = 905809408
gguf_ex_read_0: tensor[15]: name = blk.0.ffn_gate_inp.weight, size = 1441792, offset = 905820672
gguf_ex_read_0: tensor[16]: name = blk.0.ffn_gate_up_exps.weight, size = 285474816, offset = 907262464
gguf_ex_read_0: tensor[17]: name = blk.0.ffn_norm.weight, size = 11264, offset = 1192737280
gguf_ex_read_0: tensor[18]: name = blk.0.ffn_up.weight, size = 3345408, offset = 1192748544
gguf_ex_read_0: tensor[19]: name = blk.0.layer_output_scale.weight, size = 4, offset = 1196093952
gguf_ex_read_0: tensor[20]: name = blk.0.post_attention_norm.weight, size = 11264, offset = 1196093984
gguf_ex_read_0: tensor[21]: name = blk.0.post_ffw_norm.weight, size = 11264, offset = 1196105248
gguf_ex_read_0: tensor[22]: name = blk.0.post_ffw_norm_1.weight, size = 11264, offset = 1196116512
gguf_ex_read_0: tensor[23]: name = blk.0.post_ffw_norm_2.weight, size = 11264, offset = 1196127776
gguf_ex_read_0: tensor[24]: name = blk.0.pre_ffw_norm_2.weight, size = 11264, offset = 1196139040
gguf_ex_read_0: tensor[25]: name = blk.1.attn_k.weight, size = 3244032, offset = 1196150304
gguf_ex_read_0: tensor[26]: name = blk.1.attn_k_norm.weight, size = 1024, offset = 1199394336
gguf_ex_read_0: tensor[27]: name = blk.1.attn_norm.weight, size = 11264, offset = 1199395360
gguf_ex_read_0: tensor[28]: name = blk.1.attn_output.weight, size = 6488064, offset = 1199406624
gguf_ex_read_0: tensor[29]: name = blk.1.attn_q.weight, size = 6488064, offset = 1205894688
gguf_ex_read_0: tensor[30]: name = blk.1.attn_q_norm.weight, size = 1024, offset = 1212382752
gguf_ex_read_0: tensor[31]: name = blk.1.attn_v.weight, size = 4730880, offset = 1212383776
gguf_ex_read_0: tensor[32]: name = blk.1.ffn_down.weight, size = 6319104, offset = 1217114656
gguf_ex_read_0: tensor[33]: name = blk.1.ffn_down_exps.scale, size = 512, offset = 1223433760
gguf_ex_read_0: tensor[34]: name = blk.1.ffn_down_exps.weight, size = 269615104, offset = 1223434272
gguf_ex_read_0: tensor[35]: name = blk.1.ffn_gate.weight, size = 3345408, offset = 1493049376
gguf_ex_read_0: tensor[36]: name = blk.1.ffn_gate_inp.scale, size = 11264, offset = 1496394784
gguf_ex_read_0: tensor[37]: name = blk.1.ffn_gate_inp.weight, size = 1441792, offset = 1496406048
gguf_ex_read_0: tensor[38]: name = blk.1.ffn_gate_up_exps.weight, size = 285474816, offset = 1497847840
gguf_ex_read_0: tensor[39]: name = blk.1.ffn_norm.weight, size = 11264, offset = 1783322656
gguf_ex_read_0: tensor[40]: name = blk.1.ffn_up.weight, size = 3345408, offset = 1783333920
gguf_ex_read_0: tensor[41]: name = blk.1.layer_output_scale.weight, size = 4, offset = 1786679328
gguf_ex_read_0: tensor[42]: name = blk.1.post_attention_norm.weight, size = 11264, offset = 1786679360
gguf_ex_read_0: tensor[43]: name = blk.1.post_ffw_norm.weight, size = 11264, offset = 1786690624
gguf_ex_read_0: tensor[44]: name = blk.1.post_ffw_norm_1.weight, size = 11264, offset = 1786701888
gguf_ex_read_0: tensor[45]: name = blk.1.post_ffw_norm_2.weight, size = 11264, offset = 1786713152
gguf_ex_read_0: tensor[46]: name = blk.1.pre_ffw_norm_2.weight, size = 11264, offset = 1786724416
gguf_ex_read_0: tensor[47]: name = blk.2.attn_k.weight, size = 3244032, offset = 1786735680
gguf_ex_read_0: tensor[48]: name = blk.2.attn_k_norm.weight, size = 1024, offset = 1789979712
gguf_ex_read_0: tensor[49]: name = blk.2.attn_norm.weight, size = 11264, offset = 1789980736
gguf_ex_read_0: tensor[50]: name = blk.2.attn_output.weight, size = 6488064, offset = 1789992000
gguf_ex_read_0: tensor[51]: name = blk.2.attn_q.weight, size = 6488064, offset = 1796480064
gguf_ex_read_0: tensor[52]: name = blk.2.attn_q_norm.weight, size = 1024, offset = 1802968128
gguf_ex_read_0: tensor[53]: name = blk.2.attn_v.weight, size = 4730880, offset = 1802969152
gguf_ex_read_0: tensor[54]: name = blk.2.ffn_down.weight, size = 6319104, offset = 1807700032
gguf_ex_read_0: tensor[55]: name = blk.2.ffn_down_exps.scale, size = 512, offset = 1814019136
gguf_ex_read_0: tensor[56]: name = blk.2.ffn_down_exps.weight, size = 269615104, offset = 1814019648
gguf_ex_read_0: tensor[57]: name = blk.2.ffn_gate.weight, size = 3345408, offset = 2083634752
gguf_ex_read_0: tensor[58]: name = blk.2.ffn_gate_inp.scale, size = 11264, offset = 2086980160
gguf_ex_read_0: tensor[59]: name = blk.2.ffn_gate_inp.weight, size = 1441792, offset = 2086991424
gguf_ex_read_0: tensor[60]: name = blk.2.ffn_gate_up_exps.weight, size = 285474816, offset = 2088433216
gguf_ex_read_0: tensor[61]: name = blk.2.ffn_norm.weight, size = 11264, offset = 2373908032
gguf_ex_read_0: tensor[62]: name = blk.2.ffn_up.weight, size = 3345408, offset = 2373919296
gguf_ex_read_0: tensor[63]: name = blk.2.layer_output_scale.weight, size = 4, offset = 2377264704
gguf_ex_read_0: tensor[64]: name = blk.2.post_attention_norm.weight, size = 11264, offset = 2377264736
gguf_ex_read_0: tensor[65]: name = blk.2.post_ffw_norm.weight, size = 11264, offset = 2377276000
gguf_ex_read_0: tensor[66]: name = blk.2.post_ffw_norm_1.weight, size = 11264, offset = 2377287264
=== /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 ===
gguf_ex_read_0: version: 3
gguf_ex_read_0: alignment: 32
gguf_ex_read_0: data offset: 22400
gguf_ex_read_0: n_kv: 23
gguf_ex_read_0: kv[0]: key = general.architecture
gguf_ex_read_0: kv[1]: key = general.type
gguf_ex_read_0: kv[2]: key = general.sampling.top_k
gguf_ex_read_0: kv[3]: key = general.sampling.top_p
gguf_ex_read_0: kv[4]: key = general.sampling.temp
gguf_ex_read_0: kv[5]: key = general.size_label
gguf_ex_read_0: kv[6]: key = general.license
gguf_ex_read_0: kv[7]: key = general.license.link
gguf_ex_read_0: kv[8]: key = general.tags
gguf_ex_read_0: kv[9]: key = general.file_type
gguf_ex_read_0: kv[10]: key = clip.has_vision_encoder
gguf_ex_read_0: kv[11]: key = clip.vision.projection_dim
gguf_ex_read_0: kv[12]: key = clip.vision.image_size
gguf_ex_read_0: kv[13]: key = clip.vision.patch_size
gguf_ex_read_0: kv[14]: key = clip.vision.embedding_length
gguf_ex_read_0: kv[15]: key = clip.vision.feed_forward_length
gguf_ex_read_0: kv[16]: key = clip.vision.block_count
gguf_ex_read_0: kv[17]: key = clip.vision.attention.head_count
gguf_ex_read_0: kv[18]: key = clip.vision.image_mean
gguf_ex_read_0: kv[19]: key = clip.vision.image_std
gguf_ex_read_0: kv[20]: key = clip.vision.projector_type
gguf_ex_read_0: kv[21]: key = clip.vision.attention.layer_norm_epsilon
gguf_ex_read_0: kv[22]: key = general.quantization_version
gguf_ex_read_0: find key: some.parameter.string not found.
gguf_ex_read_0: n_tensors: 356
gguf_ex_read_0: tensor[0]: name = mm.input_projection.weight, size = 6488064, offset = 0
gguf_ex_read_0: tensor[1]: name = v.blk.0.ln1.weight, size = 4608, offset = 6488064
gguf_ex_read_0: tensor[2]: name = v.blk.0.ffn_down.weight, size = 9916416, offset = 6492672
gguf_ex_read_0: tensor[3]: name = v.blk.0.ffn_gate.weight, size = 9916416, offset = 16409088
gguf_ex_read_0: tensor[4]: name = v.blk.0.ffn_up.weight, size = 9916416, offset = 26325504
gguf_ex_read_0: tensor[5]: name = v.blk.0.attn_post_norm.weight, size = 4608, offset = 36241920
gguf_ex_read_0: tensor[6]: name = v.blk.0.ffn_post_norm.weight, size = 4608, offset = 36246528
gguf_ex_read_0: tensor[7]: name = v.blk.0.ln2.weight, size = 4608, offset = 36251136
gguf_ex_read_0: tensor[8]: name = v.blk.0.attn_k_norm.weight, size = 288, offset = 36255744
gguf_ex_read_0: tensor[9]: name = v.blk.0.attn_k.weight, size = 2654208, offset = 36256032
gguf_ex_read_0: tensor[10]: name = v.blk.0.attn_out.weight, size = 2654208, offset = 38910240
gguf_ex_read_0: tensor[11]: name = v.blk.0.attn_q_norm.weight, size = 288, offset = 41564448
gguf_ex_read_0: tensor[12]: name = v.blk.0.attn_q.weight, size = 2654208, offset = 41564736
gguf_ex_read_0: tensor[13]: name = v.blk.0.attn_v.weight, size = 2654208, offset = 44218944
gguf_ex_read_0: tensor[14]: name = v.blk.1.ln1.weight, size = 4608, offset = 46873152
gguf_ex_read_0: tensor[15]: name = v.blk.1.ffn_down.weight, size = 9916416, offset = 46877760
gguf_ex_read_0: tensor[16]: name = v.blk.1.ffn_gate.weight, size = 9916416, offset = 56794176
gguf_ex_read_0: tensor[17]: name = v.blk.1.ffn_up.weight, size = 9916416, offset = 66710592
gguf_ex_read_0: tensor[18]: name = v.blk.1.attn_post_norm.weight, size = 4608, offset = 76627008
gguf_ex_read_0: tensor[19]: name = v.blk.1.ffn_post_norm.weight, size = 4608, offset = 76631616
gguf_ex_read_0: tensor[20]: name = v.blk.1.ln2.weight, size = 4608, offset = 76636224
gguf_ex_read_0: tensor[21]: name = v.blk.1.attn_k_norm.weight, size = 288, offset = 76640832
gguf_ex_read_0: tensor[22]: name = v.blk.1.attn_k.weight, size = 2654208, offset = 76641120
gguf_ex_read_0: tensor[23]: name = v.blk.1.attn_out.weight, size = 2654208, offset = 79295328
gguf_ex_read_0: tensor[24]: name = v.blk.1.attn_q_norm.weight, size = 288, offset = 81949536
gguf_ex_read_0: tensor[25]: name = v.blk.1.attn_q.weight, size = 2654208, offset = 81949824
gguf_ex_read_0: tensor[26]: name = v.blk.1.attn_v.weight, size = 2654208, offset = 84604032
gguf_ex_read_0: tensor[27]: name = v.blk.10.ln1.weight, size = 4608, offset = 87258240
gguf_ex_read_0: tensor[28]: name = v.blk.10.ffn_down.weight, size = 9916416, offset = 87262848
gguf_ex_read_0: tensor[29]: name = v.blk.10.ffn_gate.weight, size = 9916416, offset = 97179264
gguf_ex_read_0: tensor[30]: name = v.blk.10.ffn_up.weight, size = 9916416, offset = 107095680
gguf_ex_read_0: tensor[31]: name = v.blk.10.attn_post_norm.weight, size = 4608, offset = 117012096
gguf_ex_read_0: tensor[32]: name = v.blk.10.ffn_post_norm.weight, size = 4608, offset = 117016704
gguf_ex_read_0: tensor[33]: name = v.blk.10.ln2.weight, size = 4608, offset = 117021312
gguf_ex_read_0: tensor[34]: name = v.blk.10.attn_k_norm.weight, size = 288, offset = 117025920
gguf_ex_read_0: tensor[35]: name = v.blk.10.attn_k.weight, size = 2654208, offset = 117026208
gguf_ex_read_0: tensor[36]: name = v.blk.10.attn_out.weight, size = 2654208, offset = 119680416
gguf_ex_read_0: tensor[37]: name = v.blk.10.attn_q_norm.weight, size = 288, offset = 122334624
gguf_ex_read_0: tensor[38]: name = v.blk.10.attn_q.weight, size = 2654208, offset = 122334912
gguf_ex_read_0: tensor[39]: name = v.blk.10.attn_v.weight, size = 2654208, offset = 124989120
gguf_ex_read_0: tensor[40]: name = v.blk.11.ln1.weight, size = 4608, offset = 127643328
gguf_ex_read_0: tensor[41]: name = v.blk.11.ffn_down.weight, size = 9916416, offset = 127647936
gguf_ex_read_0: tensor[42]: name = v.blk.11.ffn_gate.weight, size = 9916416, offset = 137564352
gguf_ex_read_0: tensor[43]: name = v.blk.11.ffn_up.weight, size = 9916416, offset = 147480768
gguf_ex_read_0: tensor[44]: name = v.blk.11.attn_post_norm.weight, size = 4608, offset = 157397184
gguf_ex_read_0: tensor[45]: name = v.blk.11.ffn_post_norm.weight, size = 4608, offset = 157401792
gguf_ex_read_0: tensor[46]: name = v.blk.11.ln2.weight, size = 4608, offset = 157406400
gguf_ex_read_0: tensor[47]: name = v.blk.11.attn_k_norm.weight, size = 288, offset = 157411008
gguf_ex_read_0: tensor[48]: name = v.blk.11.attn_k.weight, size = 2654208, offset = 157411296
gguf_ex_read_0: tensor[49]: name = v.blk.11.attn_out.weight, size = 2654208, offset = 160065504
gguf_ex_read_0: tensor[50]: name = v.blk.11.attn_q_norm.weight, size = 288, offset = 162719712
gguf_ex_read_0: tensor[51]: name = v.blk.11.attn_q.weight, size = 2654208, offset = 162720000
gguf_ex_read_0: tensor[52]: name = v.blk.11.attn_v.weight, size = 2654208, offset = 165374208
gguf_ex_read_0: tensor[53]: name = v.blk.12.ln1.weight, size = 4608, offset = 168028416
gguf_ex_read_0: tensor[54]: name = v.blk.12.ffn_down.weight, size = 9916416, offset = 168033024
gguf_ex_read_0: tensor[55]: name = v.blk.12.ffn_gate.weight, size = 9916416, offset = 177949440
gguf_ex_read_0: tensor[56]: name = v.blk.12.ffn_up.weight, size = 9916416, offset = 187865856
gguf_ex_read_0: tensor[57]: name = v.blk.12.attn_post_norm.weight, size = 4608, offset = 197782272
gguf_ex_read_0: tensor[58]: name = v.blk.12.ffn_post_norm.weight, size = 4608, offset = 197786880
gguf_ex_read_0: tensor[59]: name = v.blk.12.ln2.weight, size = 4608, offset = 197791488
gguf_ex_read_0: tensor[60]: name = v.blk.12.attn_k_norm.weight, size = 288, offset = 197796096
gguf_ex_read_0: tensor[61]: name = v.blk.12.attn_k.weight, size = 2654208, offset = 197796384
gguf_ex_read_0: tensor[62]: name = v.blk.12.attn_out.weight, size = 2654208, offset = 200450592
gguf_ex_read_0: tensor[63]: name = v.blk.12.attn_q_norm.weight, size = 288, offset = 203104800
gguf_ex_read_0: tensor[64]: name = v.blk.12.attn_q.weight, size = 2654208, offset = 203105088
gguf_ex_read_0: tensor[65]: name = v.blk.12.attn_v.weight, size = 2654208, offset = 205759296
gguf_ex_read_0: tensor[66]: name = v.blk.13.ln1.weight, size = 4608, offset = 208413504
gguf_ex_read_0: tensor[67]: name = v.blk.13.ffn_down.weight, size = 9916416, offset = 208418112
gguf_ex_read_0: tensor[68]: name = v.blk.13.ffn_gate.weight, size = 9916416, offset = 218334528
gguf_ex_read_0: tensor[69]: name = v.blk.13.ffn_up.weight, size = 9916416, offset = 228250944
gguf_ex_read_0: tensor[70]: name = v.blk.13.attn_post_norm.weight, size = 4608, offset = 238167360
gguf_ex_read_0: tensor[71]: name = v.blk.13.ffn_post_norm.weight, size = 4608, offset = 238171968
gguf_ex_read_0: tensor[72]: name = v.blk.13.ln2.weight, size = 4608, offset = 238176576
gguf_ex_read_0: tensor[73]: name = v.blk.13.attn_k_norm.weight, size = 288, offset = 238181184
gguf_ex_read_0: tensor[74]: name = v.blk.13.attn_k.weight, size = 2654208, offset = 238181472
gguf_ex_read_0: tensor[75]: name = v.blk.13.attn_out.weight, size = 2654208, offset = 240835680
gguf_ex_read_0: tensor[76]: name = v.blk.13.attn_q_norm.weight, size = 288, offset = 243489888
gguf_ex_read_0: tensor[77]: name = v.blk.13.attn_q.weight, size = 2654208, offset = 243490176
gguf_ex_read_0: tensor[78]: name = v.blk.13.attn_v.weight, size = 2654208, offset = 246144384
gguf_ex_read_0: tensor[79]: name = v.blk.14.ln1.weight, size = 4608, offset = 248798592
gguf_ex_read_0: tensor[80]: name = v.blk.14.ffn_down.weight, size = 9916416, offset = 248803200
gguf_ex_read_0: tensor[81]: name = v.blk.14.ffn_gate.weight, size = 9916416, offset = 258719616
gguf_ex_read_0: tensor[82]: name = v.blk.14.ffn_up.weight, size = 9916416, offset = 268636032
gguf_ex_read_0: tensor[83]: name = v.blk.14.attn_post_norm.weight, size = 4608, offset = 278552448
gguf_ex_read_0: tensor[84]: name = v.blk.14.ffn_post_norm.weight, size = 4608, offset = 278557056
gguf_ex_read_0: tensor[85]: name = v.blk.14.ln2.weight, size = 4608, offset = 278561664
gguf_ex_read_0: tensor[86]: name = v.blk.14.attn_k_norm.weight, size = 288, offset = 278566272
gguf_ex_read_0: tensor[87]: name = v.blk.14.attn_k.weight, size = 2654208, offset = 278566560
gguf_ex_read_0: tensor[88]: name = v.blk.14.attn_out.weight, size = 2654208, offset = 281220768
gguf_ex_read_0: tensor[89]: name = v.blk.14.attn_q_norm.weight, size = 288, offset = 283874976
gguf_ex_read_0: tensor[90]: name = v.blk.14.attn_q.weight, size = 2654208, offset = 283875264>set -e
cd ~/src/llama.cpp
MODEL="$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5"
MMPROJ="$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531"
rm -f /tmp/llama-pi/gemma-local.log /tmp/llama-pi/gemma-local.pid
nohup ./build/bin/llama-server -m "$MODEL" --mmproj "$MMPROJ" --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080 >/tmp/llama-pi/gemma-local.log 2>&1 &
echo $! >/tmp/llama-pi/gemma-local.pid
sleep 8
printf 'pid='; cat /tmp/llama-pi/gemma-local.pid
echo '--- log tail ---'
tail -n 120 /tmp/llama-pi/gemma-local.log
pid=32115
--- log tail ---
llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
llama_model_loader: - kv 5: general.size_label str = 128x2.6B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.license.link str = https://ai.google.dev/gemma/docs/gemm...
llama_model_loader: - kv 8: general.tags arr[str,1] = ["image-text-to-text"]
llama_model_loader: - kv 9: gemma4.block_count u32 = 30
llama_model_loader: - kv 10: gemma4.context_length u32 = 262144
llama_model_loader: - kv 11: gemma4.embedding_length u32 = 2816
llama_model_loader: - kv 12: gemma4.feed_forward_length u32 = 2112
llama_model_loader: - kv 13: gemma4.attention.head_count u32 = 16
llama_model_loader: - kv 14: gemma4.attention.head_count_kv arr[i32,30] = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, ...
llama_model_loader: - kv 15: gemma4.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 16: gemma4.rope.freq_base_swa f32 = 10000.000000
llama_model_loader: - kv 17: gemma4.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 18: gemma4.expert_count u32 = 128
llama_model_loader: - kv 19: gemma4.expert_used_count u32 = 8
llama_model_loader: - kv 20: gemma4.attention.key_length u32 = 512
llama_model_loader: - kv 21: gemma4.attention.value_length u32 = 512
llama_model_loader: - kv 22: gemma4.final_logit_softcapping f32 = 30.000000
llama_model_loader: - kv 23: gemma4.attention.sliding_window u32 = 1024
llama_model_loader: - kv 24: gemma4.attention.shared_kv_layers u32 = 0
llama_model_loader: - kv 25: gemma4.embedding_length_per_layer_input u32 = 0
llama_model_loader: - kv 26: gemma4.attention.sliding_window_pattern arr[bool,30] = [true, true, true, true, true, false,...
llama_model_loader: - kv 27: gemma4.attention.key_length_swa u32 = 256
llama_model_loader: - kv 28: gemma4.attention.value_length_swa u32 = 256
llama_model_loader: - kv 29: gemma4.expert_feed_forward_length u32 = 704
llama_model_loader: - kv 30: gemma4.rope.dimension_count u32 = 512
llama_model_loader: - kv 31: gemma4.rope.dimension_count_swa u32 = 256
llama_model_loader: - kv 32: tokenizer.ggml.model str = gemma4
llama_model_loader: - kv 33: tokenizer.ggml.tokens arr[str,262144] = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv 34: tokenizer.ggml.scores arr[f32,262144] = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv 35: tokenizer.ggml.token_type arr[i32,262144] = [3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 36: tokenizer.ggml.merges arr[str,514906] = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
llama_model_loader: - kv 37: tokenizer.ggml.bos_token_id u32 = 2
llama_model_loader: - kv 38: tokenizer.ggml.eos_token_id u32 = 1
llama_model_loader: - kv 39: tokenizer.ggml.unknown_token_id u32 = 3
llama_model_loader: - kv 40: tokenizer.ggml.padding_token_id u32 = 0
llama_model_loader: - kv 41: tokenizer.ggml.mask_token_id u32 = 4
llama_model_loader: - kv 42: tokenizer.chat_template str = {%- macro format_parameters(propertie...
llama_model_loader: - kv 43: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 44: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 45: general.quantization_version u32 = 2
llama_model_loader: - kv 46: general.file_type u32 = 15
llama_model_loader: - type f32: 392 tensors
llama_model_loader: - type q5_0: 32 tensors
llama_model_loader: - type q8_0: 28 tensors
llama_model_loader: - type q4_K: 192 tensors
llama_model_loader: - type q6_K: 14 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 15.63 GiB (5.32 BPW)
load: 0 unused tokens
load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load: - 1 ('<eos>')
load: - 106 ('<turn|>')
load: - 212 ('</s>')
load: special tokens cache size = 25
load: token to piece cache size = 1.9445 MB
print_info: arch = gemma4
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 2816
print_info: n_embd_inp = 2816
print_info: n_layer = 30
print_info: n_head = 16
print_info: n_head_kv = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2]
print_info: n_rot = 512
print_info: n_swa = 1024
print_info: is_swa_any = 1
print_info: n_embd_head_k = 512
print_info: n_embd_head_v = 512
print_info: n_gqa = [2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8]
print_info: n_embd_k_gqa = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: n_embd_v_gqa = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 1.0e+00
print_info: n_ff = 2112
print_info: n_expert = 128
print_info: n_expert_used = 8
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: freq_base_swa = 10000.0
print_info: freq_scale_swa = 1
print_info: n_embd_head_k_swa = 256
print_info: n_embd_head_v_swa = 256
print_info: n_rot_swa = 256
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: model type = ?B
print_info: model params = 25.23 B
print_info: general.name = n/a
print_info: vocab type = SPM
print_info: n_vocab = 262144
print_info: n_merges = 0
print_info: BOS token = 2 '<bos>'
print_info: EOS token = 1 '<eos>'
print_info: UNK token = 3 '<unk>'
print_info: PAD token = 0 '<pad>'
print_info: MASK token = 4 '<mask>'
print_info: LF token = 248 '<0x0A>'
print_info: EOG token = 1 '<eos>'
print_info: EOG token = 106 '<turn|>'
print_info: EOG token = 212 '</s>'
print_info: max token length = 93
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
str: cannot properly format tensor name output with suffix=weight bid=-1 xid=-1>for i in $(seq 1 18); do
echo "== poll $i =="
ps -p $(cat /tmp/llama-pi/gemma-local.pid) -o pid,state,etime,%cpu,%mem,command || true
lsof -nP -iTCP:8080 -sTCP:LISTEN || true
if curl -sf http://127.0.0.1:8080/v1/models >/tmp/llama-pi/models.json 2>/dev/null; then
echo '--- models ---'
cat /tmp/llama-pi/models.json
break
fi
tail -n 40 /tmp/llama-pi/gemma-local.log
sleep 10
done
echo '--- final log tail ---'
tail -n 120 /tmp/llama-pi/gemma-local.log== poll 1 ==
PID STAT ELAPSED %CPU %MEM COMMAND
32115 S 00:15 0.0 35.6 ./build/bin/llama-server -m /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 --mmproj /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 --alias gemma-4-26b-a4b-it-q4km --host 127.0.0.1 --port 8080
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
llama-ser 32115 badlogic 3u IPv4 0xd84750bbe71385d4 0t0 TCP 127.0.0.1:8080 (LISTEN)
--- models ---
{"models":[{"name":"gemma-4-26b-a4b-it-q4km","model":"gemma-4-26b-a4b-it-q4km","modified_at":"","size":"","digest":"","type":"model","description":"","tags":[""],"capabilities":["completion","multimodal"],"parameters":"","details":{"parent_model":"","format":"gguf","family":"","families":[""],"parameter_size":"","quantization_level":""}}],"object":"list","data":[{"id":"gemma-4-26b-a4b-it-q4km","aliases":["gemma-4-26b-a4b-it-q4km"],"tags":[],"object":"model","created":1775160108,"owned_by":"llamacpp","meta":{"vocab_type":1,"n_vocab":262144,"n_ctx_train":262144,"n_embd":2816,"n_params":25233142046,"size":16780192888}}]}--- final log tail ---
load_tensors: offloading output layer to GPU
load_tensors: offloading 29 repeating layers to GPU
load_tensors: offloaded 31/31 layers to GPU
load_tensors: CPU_Mapped model buffer size = 577.50 MiB
load_tensors: MTL0_Mapped model buffer size = 16002.84 MiB
....................................................................
common_init_result: added <eos> logit bias = -inf
common_init_result: added <turn|> logit bias = -inf
common_init_result: added </s> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 4
llama_context: n_ctx = 262144
llama_context: n_ctx_seq = 262144
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = true
llama_context: freq_base = 1000000.0
llama_context: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: use fusion = true
ggml_metal_init: use concurrency = true
ggml_metal_init: use graph optimize = true
llama_context: CPU output buffer size = 4.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 262144 cells
llama_kv_cache: MTL0 KV buffer size = 5120.00 MiB
llama_kv_cache: size = 5120.00 MiB (262144 cells, 5 layers, 4/1 seqs), K (f16): 2560.00 MiB, V (f16): 2560.00 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating SWA KV cache, size = 4608 cells
llama_kv_cache: MTL0 KV buffer size = 900.00 MiB
llama_kv_cache: size = 900.00 MiB ( 4608 cells, 25 layers, 4/1 seqs), K (f16): 450.00 MiB, V (f16): 450.00 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve: MTL0 compute buffer size = 792.52 MiB
sched_reserve: CPU compute buffer size = 532.02 MiB
sched_reserve: graph nodes = 2647
sched_reserve: graph splits = 2
sched_reserve: reserve took 87.92 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
clip_model_loader: model name:
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment: 32
clip_model_loader: n_tensors: 356
clip_model_loader: n_kv: 23
clip_model_loader: has vision encoder
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: use fusion = true
ggml_metal_init: use concurrency = true
ggml_metal_init: use graph optimize = true
clip_ctx: CLIP using MTL0 backend
load_hparams: projector: gemma4v
load_hparams: n_embd: 1152
load_hparams: n_head: 16
load_hparams: n_ff: 4304
load_hparams: n_layer: 27
load_hparams: ffn_op: gelu_quick
load_hparams: projection_dim: 2816
--- vision hparams ---
load_hparams: image_size: 224
load_hparams: patch_size: 16
load_hparams: has_llava_proj: 0
load_hparams: minicpmv_version: 0
load_hparams: n_merge: 3
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels: 580608
load_hparams: image_max_pixels: 645120
load_hparams: model size: 1137.77 MiB
load_hparams: metadata size: 0.12 MiB
warmup: warmup with image size = 768 x 768
alloc_compute_meta: MTL0 compute buffer size = 154.00 MiB
alloc_compute_meta: CPU compute buffer size = 6.77 MiB
alloc_compute_meta: graph splits = 1, nodes = 1569
warmup: flash attention is enabled
srv load_model: loaded multimodal model, '/Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531'
srv load_model: initializing slots, n_slots = 4
no implementations specified for speculative decoding
slot load_model: id 0 | task -1 | speculative decoding context not initialized
slot load_model: id 0 | task -1 | new slot, n_ctx = 262144
no implementations specified for speculative decoding
slot load_model: id 1 | task -1 | speculative decoding context not initialized
slot load_model: id 1 | task -1 | new slot, n_ctx = 262144
no implementations specified for speculative decoding
slot load_model: id 2 | task -1 | speculative decoding context not initialized
slot load_model: id 2 | task -1 | new slot, n_ctx = 262144
no implementations specified for speculative decoding
slot load_model: id 3 | task -1 | speculative decoding context not initialized
slot load_model: id 3 | task -1 | new slot, n_ctx = 262144
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use `--cache-ram 0` to disable the prompt cache
srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
init: chat template, example_format: '<bos><|turn>system
<|think|>You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
'
srv init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv update_slots: all slots are idleServer is up on 127.0.0.1:8080. I’m running one actual chat completion against the Gemma alias now.
>curl -sS http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gemma-4-26b-a4b-it-q4km",
"messages": [
{"role": "user", "content": "Reply with exactly: llama.cpp works"}
],
"stream": false,
"max_tokens": 32
}'{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","content":"<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"}}],"created":1775160115,"model":"gemma-4-26b-a4b-it-q4km","system_fingerprint":"b8639-a1cfb6453","object":"chat.completion","usage":{"completion_tokens":32,"prompt_tokens":23,"total_tokens":55,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-2PCO3qfNFw7nSHECJu6jAKjcIRmZjbUM","timings":{"cache_n":0,"prompt_n":23,"prompt_ms":426.618,"prompt_per_token_ms":18.548608695652174,"prompt_per_second":53.91239938305463,"predicted_n":32,"predicted_ms":545.299,"predicted_per_token_ms":17.04059375,"predicted_per_second":58.683401216580265}}The server works, but the first OpenAI chat response is garbage tokens (<unused25>...). I’m doing one more probe with simpler generation settings to see if this is just a chat-template issue.
>echo '--- /completion ---'
curl -sS http://127.0.0.1:8080/completion \
-H 'Content-Type: application/json' \
-d '{
"prompt": "Say exactly: llama.cpp works",
"n_predict": 24,
"temperature": 0,
"stop": ["\n"]
}'
echo '\n--- /v1/completions ---'
curl -sS http://127.0.0.1:8080/v1/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gemma-4-26b-a4b-it-q4km",
"prompt": "Say exactly: llama.cpp works",
"max_tokens": 24,
"temperature": 0
}'
echo '\n--- recent server log ---'
tail -n 80 /tmp/llama-pi/gemma-local.log--- /completion ---
{"index":0,"content":"<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>","tokens":[],"id_slot":2,"stop":true,"model":"gemma-4-26b-a4b-it-q4km","tokens_predicted":24,"tokens_evaluated":7,"generation_settings":{"seed":4294967295,"temperature":0.0,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":262144,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"stop":["\n"],"max_tokens":24,"n_predict":24,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"logit_bias":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"chat_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"generation_prompt":"","samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.n_max":16,"speculative.n_min":0,"speculative.p_min":0.75,"speculative.type":"none","speculative.ngram_size_n":1024,"speculative.ngram_size_m":1024,"speculative.ngram_m_hits":1024,"timings_per_token":false,"post_sampling_probs":false,"backend_sampling":false,"lora":[]},"prompt":"Say exactly: llama.cpp works","has_new_line":false,"truncated":false,"stop_type":"limit","stopping_word":"","tokens_cached":30,"timings":{"cache_n":0,"prompt_n":7,"prompt_ms":117.52,"prompt_per_token_ms":16.788571428571426,"prompt_per_second":59.5643294758339,"predicted_n":24,"predicted_ms":406.193,"predicted_per_token_ms":16.92470833333333,"predicted_per_second":59.085213186834835}}\n--- /v1/completions ---
{"choices":[{"text":"<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>","index":0,"logprobs":null,"finish_reason":"length"}],"created":1775160127,"model":"gemma-4-26b-a4b-it-q4km","system_fingerprint":"b8639-a1cfb6453","object":"text_completion","usage":{"completion_tokens":24,"prompt_tokens":7,"total_tokens":31,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-YKnITewt1Lggu0yhwBMBOcD5vKPrLXp4","timings":{"cache_n":0,"prompt_n":7,"prompt_ms":74.838,"prompt_per_token_ms":10.691142857142856,"prompt_per_second":93.53536973195436,"predicted_n":24,"predicted_ms":404.905,"predicted_per_token_ms":16.871041666666667,"predicted_per_second":59.27316284066633}}\n--- recent server log ---
slot load_model: id 2 | task -1 | speculative decoding context not initialized
slot load_model: id 2 | task -1 | new slot, n_ctx = 262144
no implementations specified for speculative decoding
slot load_model: id 3 | task -1 | speculative decoding context not initialized
slot load_model: id 3 | task -1 | new slot, n_ctx = 262144
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use `--cache-ram 0` to disable the prompt cache
srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
init: chat template, example_format: '<bos><|turn>system
<|think|>You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
'
srv init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-native
slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
slot update_slots: id 3 | task 0 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 23
slot update_slots: id 3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 19, batch.n_tokens = 19, progress = 0.826087
slot update_slots: id 3 | task 0 | n_tokens = 19, memory_seq_rm [19, end)
slot init_sampler: id 3 | task 0 | init sampler, took 0.00 ms, tokens: text = 23, total = 23
slot update_slots: id 3 | task 0 | prompt processing done, n_tokens = 23, batch.n_tokens = 4
slot print_timing: id 3 | task 0 |
prompt eval time = 426.62 ms / 23 tokens ( 18.55 ms per token, 53.91 tokens per second)
eval time = 545.30 ms / 32 tokens ( 17.04 ms per token, 58.68 tokens per second)
total time = 971.92 ms / 55 tokens
slot release: id 3 | task 0 | stop processing: n_tokens = 54, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot get_availabl: id 2 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 2 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 2 | task 34 | processing task, is_child = 0
slot update_slots: id 2 | task 34 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 7
slot update_slots: id 2 | task 34 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 2 | task 34 | prompt processing progress, n_tokens = 3, batch.n_tokens = 3, progress = 0.428571
slot update_slots: id 2 | task 34 | n_tokens = 3, memory_seq_rm [3, end)
slot init_sampler: id 2 | task 34 | init sampler, took 0.00 ms, tokens: text = 7, total = 7
slot update_slots: id 2 | task 34 | prompt processing done, n_tokens = 7, batch.n_tokens = 4
slot print_timing: id 2 | task 34 |
prompt eval time = 117.52 ms / 7 tokens ( 16.79 ms per token, 59.56 tokens per second)
eval time = 406.19 ms / 24 tokens ( 16.92 ms per token, 59.09 tokens per second)
total time = 523.71 ms / 31 tokens
slot release: id 2 | task 34 | stop processing: n_tokens = 30, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: done request: POST /completion 127.0.0.1 200
slot get_availabl: id 2 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.233
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 30, total state size = 6.447 MiB
srv load: - looking for better prompt, base f_keep = 0.233, sim = 1.000
srv update: - cache state: 1 prompts, 6.447 MiB (limits: 8192.000 MiB, 262144 tokens, 262144 est)
srv update: - prompt 0x72cc07b90: 30 tokens, checkpoints: 0, 6.447 MiB
srv get_availabl: prompt cache update took 1.64 ms
slot launch_slot_: id 2 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 2 | task 60 | processing task, is_child = 0
slot update_slots: id 2 | task 60 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 7
slot update_slots: id 2 | task 60 | n_past = 7, slot.prompt.tokens.size() = 30, seq_id = 2, pos_min = 0, n_swa = 1024
slot update_slots: id 2 | task 60 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id 2 | task 60 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 2 | task 60 | prompt processing progress, n_tokens = 3, batch.n_tokens = 3, progress = 0.428571
slot update_slots: id 2 | task 60 | n_tokens = 3, memory_seq_rm [3, end)
slot init_sampler: id 2 | task 60 | init sampler, took 0.00 ms, tokens: text = 7, total = 7
slot update_slots: id 2 | task 60 | prompt processing done, n_tokens = 7, batch.n_tokens = 4
slot print_timing: id 2 | task 60 |
prompt eval time = 74.84 ms / 7 tokens ( 10.69 ms per token, 93.54 tokens per second)
eval time = 404.90 ms / 24 tokens ( 16.87 ms per token, 59.27 tokens per second)
total time = 479.74 ms / 31 tokens
slot release: id 2 | task 60 | stop processing: n_tokens = 30, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: done request: POST /v1/completions 127.0.0.1 200Evaluating output issues
I need to provide a final, concise technical update about the server. The server is working, but it’s producing junk outputs, which might be an issue with either the model or the server itself. The user asked to try again, and we did. I should mention the command that was used and the current process ID and logs. It might be helpful to suggest the next steps, like trying a different Gemma quant/model or using the latest llama.cpp version, as there could be a bug in Gemma4.
Retried successfully with the local cached GGUFs instead of -hf.
Current state:
llama-server is runninghttp://127.0.0.1:8080/v1/models worksLaunch command that worked:
~/src/llama.cpp/build/bin/llama-server \ -m ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 \ --mmproj ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080Verified:
/v1/models returns gemma-4-26b-a4b-it-q4kmBut generation is not usable yet:
/v1/chat/completions returns repeated <unused25>/completion and /v1/completions do the sameSo the server stack is working, but this Gemma 4 GGUF combo is not producing sane text in the current llama.cpp run. That points to one of:
llama.cppFiles/logs:
/tmp/llama-pi/gemma-local.log/tmp/llama-pi/gemma-local.pidIf you want, next I can:
llama.cppwhat does the model output?
It outputs garbage placeholder tokens.
Examples:
/v1/chat/completions
{ "message": { "role": "assistant", "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>" }}/completion
{ "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"}So:
Not a port/build issue anymore. It is a model/runtime correctness issue.
ok, delete llama cpp and the model cache shit
Considering removal confirmation
I need to ask for confirmation since the user is explicitly requesting destructive removal. I might want to comply with that request directly. Should I use the bash command rm? It looks allowed here. I don’t need to read any files or modify code, so I think I can go ahead. However, I should make sure to identify paths clearly to avoid any potential mistakes!
Removing:
~/src/llama.cppggml-org/gemma-4-26b-a4b-it-GGUFI’ll stop any running llama-server first.
>set -e
PID_FILES="/tmp/llama-pi/gemma-server.pid /tmp/llama-pi/gemma-local.pid"
for pf in $PID_FILES; do
if [ -f "$pf" ]; then
pid=$(cat "$pf" 2>/dev/null || true)
if [ -n "$pid" ] && ps -p "$pid" >/dev/null 2>&1; then
kill "$pid" || true
sleep 1
if ps -p "$pid" >/dev/null 2>&1; then
kill -9 "$pid" || true
fi
fi
fi
done
rm -rf "$HOME/src/llama.cpp"
rm -rf "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF"
rm -rf /tmp/llama-pi
echo '--- remaining ---'
for p in "$HOME/src/llama.cpp" "$HOME/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF" /tmp/llama-pi; do
if [ -e "$p" ]; then
echo "still exists: $p"
else
echo "removed: $p"
fi
done--- remaining ---
removed: /Users/badlogic/src/llama.cpp
removed: /Users/badlogic/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF
removed: /tmp/llama-piRemoved:
~/src/llama.cpp~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/tmp/llama-piAny launched llama-server processes from this session were stopped first.
xcode-select --installbrew install cmakecd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j~/src/llama.cpp/build/bin/llama-serverllama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080curl http://127.0.0.1:8080/v1/modelscurl http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "gemma-4-26b-a4b-it-q4km", "messages": [ { "role": "user", "content": "say hello" } ], "stream": false }'{ "providers": { "llama-cpp": { "baseUrl": "http://127.0.0.1:8080/v1", "api": "openai-completions", "apiKey": "dummy", "compat": { "supportsDeveloperRole": false }, "models": [ { "id": "gemma-4-26b-a4b-it-q4km", "name": "Gemma 4 26B A4B IT Q4_K_M (llama.cpp local)", "reasoning": false, "input": ["text"] } ] } }}pi --model llama-cpp/gemma-4-26b-a4b-it-q4km"compat": { "supportsDeveloperRole": false, "supportsReasoningEffort": false, "supportsUsageInStreaming": false, "maxTokensField": "max_tokens"}cd ~/src/llama.cppgit pullcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q5_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080tmux new -s llama~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080xcode-select --installbrew install cmakecd ~/srcgit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_METAL=ONcmake --build build --config Release -j~/src/llama.cpp/build/bin/llama-server \ -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080{ "providers": { "llama-cpp": { "baseUrl": "http://127.0.0.1:8080/v1", "api": "openai-completions", "apiKey": "dummy", "compat": { "supportsDeveloperRole": false }, "models": [ { "id": "gemma-4-26b-a4b-it-q4km" } ] } }}~/src/llama.cpp/build/bin/llama-server \ -m ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/23c6997912cb7fa36147fe05877de73ddbb2a80ff69b18ff171b354dccf2b5b5 \ --mmproj ~/.cache/huggingface/hub/models--ggml-org--gemma-4-26b-a4b-it-GGUF/blobs/4107c1c3c299095fbc323f87f4e4cac81dd9527db5ff90808fea669e08244531 \ --alias gemma-4-26b-a4b-it-q4km \ --host 127.0.0.1 \ --port 8080{ "message": { "role": "assistant", "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>" }}{ "content": "<unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25><unused25>"}