Serve GLM-5.3 Flash · FP8 privately, connect Hermes to the model endpoint, and have the agent inspect files and produce a review checklist.
Complete checkpoint plus runtime and KV-cache capacity, not active parameters or a speed score.
| Machine | GLM-5.3 Flash · FP8 |
|---|---|
| 4× GB200 · at least 768 GB aggregate GPU memory | Documented starting class; begin at 16,384 tokens and confirm free memory. |
| Single RTX 5090 · 32 GB VRAM | Does not hold this full checkpoint in the documented server route; choose 4× GB200. |
| AMD GPU server | The NVIDIA command is unsupported on AMD; use the publisher’s separate 4× MI355X ROCm recipe. |
| Intel GPU server | No documented Intel serving route for this checkpoint; use the listed NVIDIA topology. |
| Apple Silicon Mac · 128 GB unified memory | No publisher native route for this package; use Mac as a client to 4× GB200. |
| CPU only · 128 GB RAM | No practical publisher CPU route; use 4× GB200 for this setup. |
The 4×MI355X AMD path needs the separate ROCm recipe; the NVIDIA command here is not interchangeable.
Run the exact checkpoint on a Linux GPU host, verify tool calling, then connect Hermes and produce a useful file-based review. Own or rent the GPU.
This setup uses the package documented for this task. Review its source and supported platforms before starting.
Save your machine in My Hardware to get an automatic starting choice. You can always choose any package yourself.
Start at 16,384 tokens on the listed GPU topology. Longer context raises KV-cache memory; lower context before changing model or renting more GPUs.
Choose your computer first, then its GPU path. The instructions below update to match your selection.
Windows: The publisher server recipe is a Linux accelerator deployment; use Windows as a private client over an SSH tunnel, not as an undocumented native inference host. Compatibility source
Mac: This publisher server recipe requires the listed Linux accelerator topology; use macOS as a private client or choose a different model for local Mac inference. Compatibility source
Use your own Linux server with 4× GB200 or compatible accelerator memory. Confirm the driver/runtime and storage before downloading the full GLM-5.3 Flash · FP8 checkpoint; active-parameter count is not a memory requirement.
Model weights and requirementsIn Linux Terminal, install Docker and pull the current vLLM CUDA nightly image required by the GLM-5.3 Flash recipe. A plain pip install is not the documented route for its sparse-MLA integration.
Publisher serving recipe Install Docker EngineIn Linux Terminal, select the publisher checkpoint and launch with the model-specific parser, GPU topology and a conservative 16,384-token context. Keep this terminal running; the model download can be large. Do not expose unauthenticated port 8000 to the public internet.
Exact official model repositoryIn a second Linux Terminal, check the model list at http://127.0.0.1:8000/v1/models. It must return the exact model ID used in the next step. For a remote client, tunnel port 8000 over SSH and keep the public port closed.
API response formatIn a second Linux Terminal, run the small read-only check against zai-org/GLM-5.3-Flash. This only proves that the server parser returns a tool call; it is not the agent setup or its final result.
Download the tool-call check Publisher tool-calling documentationOn Windows or macOS, use the official Hermes Desktop installer. On Linux, use the official command-line installer. Keep the model server private and make an SSH tunnel from your computer to port 8000; do not publish an unauthenticated model API.
Hermes installers for Windows, macOS, and Linux Hermes custom endpoint setupOn your computer, run hermes model and choose Custom endpoint. Set the API base URL to http://127.0.0.1:8000/v1 through your SSH tunnel, enter zai-org/GLM-5.3-Flash as the model name, and use a context no larger than 16,384 tokens. Check /v1/models first; if Hermes cannot make a tool call, stop here rather than treating plain chat as an agent.
Official custom endpoint instructions vLLM endpoint formatCreate a disposable project folder containing a launch note and a short checklist with one deliberately missing backup step. In Hermes, ask the agent to inspect those files and create review.md with a corrected checklist, the missing backup risk, and source filenames. Approve file access and any write only after reviewing the proposed action. This is the workflow outcome; the earlier tool-call probe is only a connection test.
Hermes agent and tool setupOpen review.md in the disposable folder. Confirm the launch day and missing-backup risk came from the files, that the corrected checklist is actionable, and that Hermes used its file tools rather than inventing an answer. Review the server log for parser errors. Stop the rental or server when done.
Hermes read the disposable source files and wrote review.md with a corrected checklist and the missing-backup risk. Inspect its tool activity and the actual file; the preliminary read_notes probe is not the final outcome.
Publisher-source-backed setup, not a YouRunAI hands-on benchmark of this model in Hermes or Cline. Verify tool compatibility on the exact runtime before relying on it. Use the vLLM CUDA nightly image required by the linked recipe. For AMD, use its separate ROCm command rather than the NVIDIA tab.
Confirm every listed GPU is visible, check full weight plus KV-cache memory, start at 16,384 tokens, then use the publisher-specific container or build before renting a larger topology.
Check the exact parser flags and vLLM compatibility in the linked publisher recipe. Plain chat is not a passing agent result; keep the model off agent recommendations until a tool call works.
Keep the service bound to a private network and use an SSH tunnel to port 8000. Check /v1/models through the tunnel; do not open an unauthenticated public endpoint.
Original instructions, model files, and compatibility notes behind this setup.
Source-backed package and platform-route plan for private local chat with Qwen3.5 · 9B. Preserves the official Ollama 9B package, a smaller 4B family fallback, and a clearly labeled community uncensored GGUF option.
Run Qwen3.8 27B with Ollama, connect it to Cline in VS Code, and make one reviewable code change in a tiny local project.
Generate images from text prompts in ComfyUI with the fast distilled or more adjustable base 4B model. Save your results and reuse the visual workflow.