Run the publisher’s RavenX Q4_K_M GGUF locally on Windows, macOS, or Linux, or on a rented RunPod GPU. Serve it with llama.cpp and connect an optional agent to a read-only security-review task.
Memory-based estimate for the publisher’s Q4_K_M package. The reported 24 GB peak is from an M4 Max; all GPU rows below are estimates, not benchmark results.
| Machine | RavenX CyberAgent · 35B-A3B · Q4_K_M |
|---|---|
| NVIDIA GPU · 24 GB VRAM (local or rented) | Tight at 8K context; 32 GB is the more practical starting point. |
| NVIDIA GPU · 32 GB VRAM (local or rented) | Estimated practical starting point at 8K local or 32K rented context. |
| NVIDIA GPU · 48 GB VRAM or more | More memory headroom for context and other GPU work. |
| AMD GPU · 32 GB VRAM | Memory may fit; use a matching ROCm or Vulkan llama.cpp build. Runtime support varies. |
| Intel Arc GPU · 32 GB VRAM | Memory may fit; use a matching Vulkan or SYCL llama.cpp build. Runtime support varies. |
| Apple Silicon · 32 GB unified memory | Tight starting point; close other apps and keep context short. |
| Apple Silicon · 48 GB unified memory or more | More comfortable estimated starting point. |
| CPU-only · 64 GB system RAM | Can hold the package in memory; expect slow generation. |
The AMD, Intel, and CPU rows depend on a compatible llama.cpp build and are not tested speed claims. Check available system memory and runtime support before downloading.
Start with 8K context locally, or 32K on the rented Pod instructions. Larger context consumes more memory.
| Machine | RavenX CyberAgent · 35B-A3B · Q4_K_M |
|---|---|
| Local GPU · 24 GB VRAM | Start at 4K–8K; tight headroom. |
| Local GPU · 32 GB VRAM or more | Start at 8K; increase only after checking free memory. |
| Apple Silicon · 32 GB unified memory | Start at 4K; close other memory-heavy apps. |
| Apple Silicon · 48 GB unified memory or more | Start at 8K; increase gradually if memory remains available. |
| CPU-only · 64 GB system RAM | Start at 4K; expect slow generation. |
| Rented GPU · 32 GB VRAM | 32K is the guide’s estimated starting setting; reduce it if loading fails. |
| Rented GPU · 48 GB VRAM or more | 32K starting setting; increase only after checking memory headroom. |
These are conservative starting settings, not measured performance claims.
Choose Windows, Mac, Linux, or RunPod. Serve the publisher’s Q4_K_M GGUF with llama.cpp, then connect an optional Hermes agent. The example task reviews one local configuration file read-only; it does not scan or exploit a live target.
This setup uses the package documented for this task. Review its source and supported platforms before starting.
Save your machine in My Hardware to get an automatic starting choice. You can always choose any package yourself.
Start at 8K context locally. 24 GB GPU memory or 32 GB Apple unified memory is tight; 32 GB VRAM or 48 GB unified memory is a more practical starting point. A CPU-only setup needs ample system RAM and will be slow. These are memory estimates, not speed tests.
Choose your computer first, then its GPU path. The instructions below update to match your selection.
In PowerShell, use the matching Windows CUDA 12 or 13 build for your operating system and GPU from the official release or installation instructions. Keep the extracted program and its required runtime libraries together. On the first launch, llama.cpp downloads the publisher’s Q4_K_M weights from Hugging Face.
Official Windows CUDA releases and CUDA libraries llama.cpp backend support RavenX publisher instructionsStart with an 8K context to leave more memory for the operating system and runtime. The server is bound to 127.0.0.1 so it is reachable only from this computer; do not change it to 0.0.0.0 for an unprotected test server.
.\llama-server.exe -hf deadbydawn101/RavenX-CyberAgent-Qwen3.6-35B-A3B-Opus-4.7-OpenMythos-Pentester-BugHunter-RATH-GGUF:Q4_K_M -ngl 99 -c 8192 --host 127.0.0.1 --port 8080Open http://127.0.0.1:8080/v1/models on this computer. It should list the served RavenX model. A compatible local client can use http://127.0.0.1:8080/v1; use a read-only, authorized sample for the first security review.
Connect Hermes or another OpenAI-compatible client to http://127.0.0.1:8080/v1 and select the model ID returned by /v1/models. Give it one disposable configuration or code file you own, allow read-only access, and ask it to return findings with exact evidence. Keep approval enabled and do not grant network scanning or write tools.
Hermes Agent setup and model configurationGET /v1/models returns the served RavenX model when called with the API key. A short request to /v1/chat/completions returns a response based on the sample configuration. For the optional Hermes run, confirm the agent read only the scoped file and saved the findings report. Independently verify every security claim.
Use only on assets you own or have explicit written authorization to assess. Keep the first agent run read-only and scoped. RunPod HTTP proxy ports are publicly accessible; use a strong API key and stop the Pod when finished. Publisher-reported model size and memory may change; check the linked source and live RunPod pricing before renting.
Confirm the selected Pod has the expected GPU and visible VRAM with nvidia-smi. Keep context at 32K for the first run, close other GPU processes, and move from 24 GB to a 32 GB or 48 GB GPU if the package plus runtime and cache do not fit. The published M4 Max result does not establish GPU performance.
Confirm llama.cpp is still running on port 8080, use the Pod-specific HTTPS proxy URL with /v1, and send the same API key configured on the server. RunPod documents a 100-second HTTP proxy timeout; long requests may need a direct client/connection path or a shorter task.
Keep Hermes and llama.cpp on the same Pod for the localhost route, use http://127.0.0.1:8080/v1, and copy the exact model ID reported by GET /v1/models. Check that the API key matches without putting it in a command or shell history.
Stop the run. Remove network and write-capable tools, keep human approval enabled, and use only a disposable lab or files you are explicitly authorized to assess. Verify every claim against the actual evidence before reporting it.
Original instructions, model files, and compatibility notes behind this setup.
Source-backed package and platform-route plan for private local chat with Qwen3.5 · 9B. Preserves the official Ollama 9B package, a smaller 4B family fallback, and a clearly labeled community uncensored GGUF option.
Run Qwen3.8 27B with Ollama, connect it to Cline in VS Code, and make one reviewable code change in a tiny local project.
Generate images from text prompts in ComfyUI with the fast distilled or more adjustable base 4B model. Save your results and reuse the visual workflow.