Ask your own documents with local AI
Pair Qwen3.5 in Ollama with Open WebUI’s document retrieval, then check that answers cite your uploaded file.
Before you begin
- Difficulty
- Intermediate
- Software
- Ollama, Open WebUI, Python 3.11
- Hardware
- The 9B Ollama package is 6.6 GB. Document retrieval, embeddings, and longer context require additional memory and disk space.
Sources and files
Open WebUI quick start Open WebUI document retrieval Qwen3.5 Ollama packagesChoose a model for this task
The steps below use the recommended model. Alternatives have their own package and command; open their model pages before switching.
The workflow
Install the local model runtime
Install Ollama for your operating system and start its local service. Use a machine that has room for both the 9B package and the document-processing stack.
Download Qwen3.5 9B
Run the published Ollama package once so the model is available to Open WebUI locally.
ollama run qwen3.5:9bInstall Open WebUI
Use a Python 3.11 environment, which Open WebUI identifies as its most tested Python version. Install the application from its official package.
pip install open-webuiStart the private interface
Start Open WebUI and open its local address in a browser. Choose the installed Ollama model in the chat model selector.
open-webui serveUpload a small source document
In Open WebUI Workspace → Documents, upload a text-based document you own. Wait for its processing to finish before asking questions.
Ask a question grounded in the file
In a chat, type #, select the uploaded document, and ask a question whose answer appears clearly in that file. The document icon should appear by the send button.
Check the answer and context
Confirm the answer is supported by the source document. If it cannot retrieve details, reduce the file scope or increase available context; Ollama can default to 4K context on GPUs below 24 GiB.
When it doesn’t go to plan
Confirm Ollama is running, the model downloaded, and Open WebUI is connected to the local Ollama endpoint. The official quick start documents connection settings.
Wait for document processing, select it with # in the chat, and verify the document icon is attached. Check the embedding model and context settings using Open WebUI’s RAG troubleshooting guide.
The model behind this workflow
Qwen3.5 · 9BWill it run on your machine?
Save your machine to see a personalized rating and its reasoning.
Add my hardwareKeep exploring
Set up a private AI chat assistant
Install a local runtime, run Qwen3.5 9B, confirm responses, and know when to choose the smaller 4B package.
Build a private local coding assistant
Connect an open coding-capable model in Ollama to Cline, run a small repository task, and review the result locally.
Generate images on your own GPU
Use the official FLUX.2 Klein 4B ComfyUI template with exact model files, a first prompt, and an output check.