ggml-metal.metal not found

[jonashaag]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

pixi run python -m llama_cpp.server ... --n_gpu_layers 1

ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x600001752520 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
llama_new_context_with_model: ggml_metal_init() failed

Fixed by copying ggml-metal.metal into CWD:

cp /path/to/conda/env/bin/ggml-metal.metal .
pixi run python -m llama_cpp.server ...

Installed packages

...
llama-cpp-python                          0.2.24             py312h20a0b95_0              166.9 KiB  conda  llama-cpp-python-0.2.24-py312h20a0b95_0.conda
llama.cpp                                 0.0.1660           mps_h18ddfd1_0               1.3 MiB    conda  llama.cpp-0.0.1660-mps_h18ddfd1_0.conda
...

Environment info

Pixi version: 0.23.0
          Platform: osx-arm64
  Virtual packages: __unix=0=0
                  : __osx=14.5=0
                  : __archspec=1=m1
         Cache dir: /Users/j/Library/Caches/rattler/cache
      Auth storage: /Users/j/.rattler/credentials.json

Environments
------------
       Environment: default
          Features: default
          Channels: conda-forge
  Dependency count: 57
                    ...

[frankier]

Is the bin directory in your $PATH? You might need to activate the Conda environment for it to work.

I'm not sure how pixi works. Does it actually activate the relevant environment including modifying the $PATH in that example? Maybe it should? Perhaps an issue could be opened with pixi?

This could be made more robust upstream by having llama_cpp.server make sure the directory of the executable llama_cpp.server was added to the end of the path. Perhaps an issue could be opened with llama_cpp.

Let's figure out whether it can be fixed in those places first, and then if not then depending on the reason you could make the workaround in this repo.

[jonashaag]

It's not a bug in Pixi. But maybe we can fix it upstream yes.

[frankier]

This seems to be the upstream issue: ggml-org/llama.cpp#5376

Perhaps we should build with LLAMA_METAL_EMBED_LIBRARY? I'm not sure.