ggml-org/llama.cpp

LLM inference in C/C++

Agents in llama.cpp

Llama.cpp with the Rockchip NPU integration as a GGML backend.

Falcon LLM ggml framework with CPU and GPU support

llama.cpp fork with TQ3_1S/4S CUDA kernels — 3.5-bit WHT quantization achieving Q4s quality at 10% smaller size. Based on RaBitQ-inspired Walsh-Hadamard transform. Enables 27B models on 16GB GPUs with 15 tok/s TG, 221 tok/s PP.

LLM inference in C/C++

Fork of llama.cpp, supporting Facebook's LLaMA model in Swift

Port of Facebook's LLaMA (Large Language Model Meta AI) in Golang with embedded C/C++

LLM inference in C/C++

LLM inference in C/C++

llama.cpp-gfx906

QVAC Fabric: cross-platform LLM inference and fine-tuning, optimized for edge devices and heterogenous GPUs

My develoopment fork of llama.cpp. For now working on RK3588 NPU and Tenstorrent backend

Extend the original llama.cpp repo to support redpajama model.

Port of Facebook's LLaMA model in C/C++

CodeShell model in C/C++

LLM inference in C/C++ on Pi Zero 1

Port of Facebook's LLaMA model in C/C++

LLM inference in C/C++

llama.cpp fork used by GPT4All

LLM inference in C/C++

LLM inference in C/C++

After my server ui improvements were successfully merged, consider this repo a playground for experimenting, tinkering and hacking around the ggml library, especially llama.cpp – and primarily its server. The best fruits will hopefully be merged upstream – as long as they are consistent with the philosophy of llama.cpp

the original reference implementation of a specified llama.cpp backend for Qualcomm Hexagon NPU on Android phone, history of ggml-hexagon: https://github.com/zhouwg/ggml-hexagon/discussions/18.

Proof of concept: Exploiting temporal coherence in LLM inference-- delta encoding for KV cache compression and weight-skip prediction. Achieves F16-quality KV cache at Q4_0 compression ratios with zero perplexity loss on llama.cpp.

Port of Facebook's LLaMA model in C/C++

LLM inference in C/C++

LLM inference in C/C++

LLM inference in C/C++

LLM inference in C/C++

Port of Facebook's LLaMA model in C/C++

VENDORIZED in lucebox-hub. Fork of llama.cpp, ggml graph for lucebox inference engine

Thireus's fork of llama.cpp with Cuda 12.8 and 13.3 release builds and Windows patch for loading more .gguf shards + llama-sweep-bench

LLM inference in C/C++

Spezi LLM inference in C/C++

Fork of llama.cpp, extended for GPT-NeoX, RWKV-v4, and Falcon models

Android JNI for port of Facebook's LLaMA model in C/C++

Port of Facebook's LLaMA model in C/C++

LLM inference in C/C++

HIP/ROCm fork optimized for AMD RDNA2 (gfx1030) with PrismML Q1_0_G128 1-bit quant support, RotorQuant, TurboQuant, EAGLE3 and P-EAGLE speculative decoding, and full Wave32 kernel optimizations.

LLM inference in C/C++

LLM inference in C/C++, but for GFX906!

LLM inference in C/C++

Port of Facebook's LLaMA model in C/C++

LLM inference in C/C++

LLM inference in C/C++