Serve Mellum2.1 Thinking · 12B BF16 privately, connect VS Code and Cline, then review and test a real code change.
Complete checkpoint plus runtime and KV-cache capacity, not active parameters or a speed score.
| Machine | Mellum2.1 Thinking · Q4_K_M GGUF | Mellum2.1 Thinking · 12B BF16 |
|---|---|---|
| NVIDIA 48 GB GPU · at least 48 GB aggregate GPU memory | Official Q4 package may fit with short context; check free VRAM and Cline tool use. | Documented starting class; begin at 16,384 tokens and confirm free memory. |
| Single RTX 5090 · 32 GB VRAM | Q4 is the easier local route; BF16 requires more memory. | Does not hold this full checkpoint in the documented server route; choose NVIDIA 48 GB GPU. |
| AMD Radeon · 16 GB VRAM | The 8.1 GB Q4 file may fit at 8K context; verify Ollama GPU offload on the exact card and driver. | No documented AMD ROCm route for this exact checkpoint; use the listed NVIDIA topology. |
| Intel Arc · 16 GB VRAM | The 8.1 GB Q4 file may fit at 8K context; verify Ollama Intel backend support and GPU offload. | No documented Intel serving route for this checkpoint; use the listed NVIDIA topology. |
| Apple Silicon Mac · 128 GB unified memory | CPU offload may work but coding edits can be slow. | No publisher native route for this package; use Mac as a client to NVIDIA 48 GB GPU. |
| CPU only · 128 GB RAM | CPU offload may work but coding edits can be slow. | No practical publisher CPU route; use NVIDIA 48 GB GPU for this setup. |
JetBrains now also publishes an 8.1 GB Q4_K_M GGUF for Windows, macOS and Linux; use that desktop route first when it fits.
Start with the official 8.1 GB Q4 GGUF on Windows, Mac or Linux and connect it to VS Code + Cline. Choose the full-precision Linux GPU route when you need it; verify a real code edit either way.
Select the weights you want. The instructions, downloads and settings change with your choice.
Save your machine in My Hardware to get an automatic starting choice. You can always choose any package yourself.
The 8.1 GB file needs additional memory for Ollama, context, and editor tasks. Start at 8,192 tokens; tool-use quality in Cline still needs a real run.
Choose your computer first, then its GPU path. The instructions below update to match your selection.
On Windows, install Ollama from its official download page. The Q4_K_M file is 8.1 GB; leave additional GPU or unified memory for the runtime and a short context. Start at 8,192 tokens rather than assuming the full 131K advertised window will fit.
Download Ollama Official JetBrains GGUF and exact file sizesIn PowerShell, start the publisher's exact Q4_K_M tag. Ollama manages its model files automatically, so there is no manual folder layout. Ask one short coding question, then leave Ollama running as the local API server.
JetBrains Ollama commandInstall VS Code and its Cline extension. In Cline choose the OpenAI-compatible provider, Base URL http://127.0.0.1:11434/v1 and Model ID hf.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M. Set an 8,192-token starting context, then confirm the exact model answers through the local endpoint. Keep approvals on for file writes and shell commands.
Download VS Code Install Cline Cline OpenAI-compatible setupOpen a disposable repository in VS Code. Ask Cline to add a simple counter button and a test. Review the proposed diff and commands before approval, run the test, and open the page to verify the button. If this quantization cannot use Cline tools reliably, stop and try the BF16 vLLM package or another tested coding model; do not count a chat answer as a passing coding setup.
Confirm the edited files, passing test and working UI in the disposable project. Record the package and available memory if you share a result; this site has not benchmarked Cline tool reliability on the Q4 package.
Cline edited the disposable repository, you approved and inspected the diff, and its test and UI behavior pass. A file-reading tool probe is only a connection check.
Publisher-source-backed setup, not a YouRunAI hands-on benchmark of this model in Hermes or Cline. Verify tool compatibility on the exact runtime before relying on it. Use VS Code and Cline for an actual reviewed code change. Validate tool use on the selected quantization.
Confirm every listed GPU is visible, check full weight plus KV-cache memory, start at 16,384 tokens, then use the publisher-specific container or build before renting a larger topology.
Check the exact parser flags and vLLM compatibility in the linked publisher recipe. Plain chat is not a passing agent result; keep the model off agent recommendations until a tool call works.
Keep the service bound to a private network and use an SSH tunnel to port 8000. Check /v1/models through the tunnel; do not open an unauthenticated public endpoint.
Original instructions, model files, and compatibility notes behind this setup.
Source-backed package and platform-route plan for private local chat with Qwen3.5 · 9B. Preserves the official Ollama 9B package, a smaller 4B family fallback, and a clearly labeled community uncensored GGUF option.
Run Qwen3.8 27B with Ollama, connect it to Cline in VS Code, and make one reviewable code change in a tiny local project.
Generate images from text prompts in ComfyUI with the fast distilled or more adjustable base 4B model. Save your results and reuse the visual workflow.