Run the official MiniCPM5 2B GGUF on Windows, Mac, or Linux with Ollama for a simple private chat. An advanced GPTQ/vLLM route is available for Linux NVIDIA, plus a separately labelled community uncensored MLX package for Apple Silicon.
Starting memory guidance for three distinct packages. The official GGUF works across Windows, Mac, and Linux; GPU acceleration depends on the supported backend.
| Machine | MiniCPM5 2B · official GGUF Q4_K_M | MiniCPM5 2B GPTQ (AWQ-packed publisher) | MiniCPM5 2B Abliterated Uncensored MLX 4-bit |
|---|---|---|---|
| NVIDIA GPU · 8 GB (Windows or Linux) | Runs locally; 8 GB GPU is a comfortable start for the 1.56 GB GGUF. | Supported: vLLM AWQ GEMM route; 2.10 GB weights; 8 GB VRAM min, 16 GB for longer context. | Unsupported here: MLX package needs Apple Silicon; use publisher AWQ package on Linux NVIDIA. |
| AMD GPU · 8 GB (Windows or Linux) | Memory likely fits; verify Ollama AMD GPU support or expect CPU offload. | Not covered: no evidence vLLM AutoAWQ supports this AMD route; use Linux NVIDIA vLLM AWQ. | Unsupported: MLX package requires Apple Silicon; no AMD MLX route in this guide. |
| Intel Arc GPU · 8 GB (Windows or Linux) | Memory likely fits; verify Ollama Intel GPU support or expect CPU offload. | Not covered: no evidence vLLM AWQ supports Intel Arc here; use Linux NVIDIA vLLM AWQ. | Unsupported: MLX package requires Apple Silicon; no Intel Arc route in this guide. |
| Apple Silicon · 8–16 GB unified memory | Runs locally with Ollama/Metal; 8 GB can work, 16 GB offers more headroom. | Unsupported: AWQ vLLM package is Linux NVIDIA only; use the MLX package here. | Supported: mlx-lm on macOS 14+, native Python 3.10+; 4-bit MLX package. |
| CPU only · 16 GB system RAM | Runs locally in system RAM; CPU-only replies may be slower. | Not covered: AWQ package is vLLM Linux NVIDIA route; no CPU AWQ route in this guide. | Unsupported: MLX package is Apple Silicon only; no CPU route in this guide. |
Start with an 8K active context on the official GGUF. Raise it only after confirming the model loads without heavy offload or swap.
| Machine | MiniCPM5 2B · official GGUF Q4_K_M | MiniCPM5 2B GPTQ (AWQ-packed publisher) | MiniCPM5 2B Abliterated Uncensored MLX 4-bit |
|---|---|---|---|
| NVIDIA GPU · 8 GB (Windows or Linux) | Start at 8K; reduce to 4K if VRAM is tight. | Start 8192 tokens (model max 131072); KV ~42 KB/token FP16 estimate; 16 GB rec. for longer chat. | No supported route here: MLX package needs Apple Silicon; use publisher AWQ package. |
| AMD GPU · 8 GB (Windows or Linux) | Start at 8K; inspect the actual GPU/CPU split. | No supported route: no AMD AWQ evidence; use Linux NVIDIA vLLM AWQ package. | No supported route: MLX package needs Apple Silicon; no AMD route in this guide. |
| Intel Arc GPU · 8 GB (Windows or Linux) | Start at 8K; inspect the actual GPU/CPU split. | No supported route: no Intel Arc AWQ evidence; use Linux NVIDIA vLLM AWQ. | No supported route: MLX package needs Apple Silicon; no Intel Arc route. |
| Apple Silicon · 8–16 GB unified memory | Start at 8K; lower it if macOS starts swapping. | No supported route: AWQ vLLM package is Linux NVIDIA only; use the MLX package. | Start 8192 tokens (model max 131072); 8 GB min, 16 GB rec.; raise after success. |
| CPU only · 16 GB system RAM | Start at 4K–8K; longer context increases RAM use and delay. | No supported route: AWQ package is vLLM Linux NVIDIA; no CPU route here. | No supported route: MLX package is Apple Silicon only; no CPU route here. |
Choose the official cross-platform GGUF for the quickest local chat, or select the Linux NVIDIA GPTQ/vLLM or Apple Silicon community uncensored MLX route. Package selection changes the exact commands below.
Select the weights you want. The instructions, downloads and settings change with your choice.
Save your machine in My Hardware to get an automatic starting choice. You can always choose any package yourself.
Publisher GGUF runs in standard llama.cpp-based tools on Windows, Mac, and Linux. A 1.56 GB file is only the model weight size; start at 8K context and check actual GPU/CPU loading in ollama ps. AMD and Intel acceleration depend on a supported driver/backend.
Choose where you want to run it. The instructions below update to match your selection.
Official 1.56 GB publisher Q4_K_M GGUF through Ollama. Start at 8K active context; verify the GPU/CPU split with ollama ps.
Download the macOS Ollama app from the official page, move it to Applications, and open it once. Open Terminal and confirm the ollama command works before running the model. Ollama uses Metal acceleration on supported Apple Silicon Macs.
Download Ollama for macOS Ollama macOS guideollama --versionOpen your system terminal and run the exact publisher GGUF through Ollama. Ollama downloads the 1.56 GB Q4_K_M file automatically; no manual folder or conversion is needed. Keep this terminal open for the chat session.
Publisher MiniCPM5 2B GGUF filesollama run hf.co/openbmb/MiniCPM5-2B-GGUF:Q4_K_MInside the interactive Ollama chat, enter /set parameter num_ctx 8192. This selects an 8K active context; the model’s larger advertised maximum is not a sensible default for every machine. If memory is tight, choose 4096. This is a chat command, not a shell command.
/set parameter num_ctx 8192Leave the chat open and use a second terminal. Confirm the exact MiniCPM5 GGUF tag is installed and that Ollama reports whether it is using the GPU or CPU. CPU use can be slower, especially at longer context.
Ollama command referenceollama list
ollama psIn the Ollama chat, ask “What is 2+2? Answer with one number.” Confirm it answers 4. Then ask “What question did I just ask?” and confirm it refers to the first turn. Your chat stays on the local Ollama service.
Ask “What is 2+2? Answer with one number.” and confirm 4, then ask a follow-up in the same local chat. For the official GGUF, verify the exact Ollama tag and GPU/CPU split with ollama list and ollama ps. The Linux GPTQ/vLLM and community uncensored MLX packages remain separate routes.
The official GGUF is the simplest starting point on Windows, Mac, and Linux. GPTQ/AWQ with vLLM is an advanced Linux NVIDIA route; the uncensored MLX package is a separate community Mac variant. Package size alone is not a speed test.
Confirm the exact config.json reports quant_method awq and format gemm. vLLM AutoAWQ support is required; do not substitute a GPTQ or GGUF file. If the error mentions unsupported quantization, verify the vLLM version includes AWQ GEMM support.
Confirm macOS 14 or newer and a native arm64 Python 3.10 or newer, as required by the MLX install documentation and mlx-lm. Reinstall mlx-lm in an isolated environment if the interpreter is not native.
Lower the launch context to a conservative value such as 8192 tokens, keep temperature 1.0 and top_p 0.95, and resend the known-answer prompt. Increase context only after a successful response.
Original instructions, model files, and compatibility notes behind this setup.
Source-backed package and platform-route plan for private local chat with Qwen3.5 · 9B. Preserves the official Ollama 9B package, a smaller 4B family fallback, and a clearly labeled community uncensored GGUF option.
Run Qwen3.8 27B with Ollama, connect it to Cline in VS Code, and make one reviewable code change in a tiny local project.
Generate images from text prompts in ComfyUI with the fast distilled or more adjustable base 4B model. Save your results and reuse the visual workflow.