VLM/LLM Serving Guide#
This guide shows how to choose the text-generation model for Smart Classroom and how to turn on the optional DFlash accelerator.
All settings are under models.text_gen in config.yaml. Edit them on the desktop
app’s Configuration screen under Models and devices, or select Show file to
open %LOCALAPPDATA%\SmartClassroom\config.yaml. Restart the backend after you change
them.
Recommended Configuration#
Pick the row that matches your platform and system memory:
Platform |
Memory |
Model |
Accelerator |
|---|---|---|---|
Arrow Lake (ARL) |
32 GB |
Qwen3.5-9B INT4 |
DFlash, |
Arrow Lake (ARL) |
64 GB |
Qwen3.6-35B-A3B INT4 |
None |
Panther Lake (PTL) |
32 GB |
Qwen3.5-9B INT4 |
DFlash, |
Panther Lake (PTL) |
64 GB |
Qwen3.6-35B-A3B INT4 |
DFlash, |
Copy the matching block into config.yaml. Keep the other keys under
models.text_gen as they are.
ARL or PTL with 32 GB: Qwen3.5-9B INT4 with DFlash
models:
text_gen:
provider: vlm
vlm_name: Qwen/Qwen3.5-9B
device: GPU
weight_format: int4
speculative:
mode: "on"
method: dflash
draft_source: null
draft_model: null
draft_revision: null
num_assistant_tokens: 7
ARL with 64 GB: Qwen3.6-35B-A3B INT4
models:
text_gen:
provider: vlm
vlm_name: Qwen/Qwen3.6-35B-A3B
device: GPU
weight_format: int4
speculative:
mode: "off"
PTL with 64 GB: Qwen3.6-35B-A3B INT4 with DFlash
models:
text_gen:
provider: vlm
vlm_name: Qwen/Qwen3.6-35B-A3B
device: GPU
weight_format: int4
speculative:
mode: "on"
method: dflash
draft_source: null
draft_model: null
draft_revision: null
num_assistant_tokens: 5
Available Models#
|
|
Optional accelerator |
|---|---|---|
|
|
None |
|
|
DFlash |
|
|
DFlash |
|
|
None |
For a model without an accelerator, set speculative.mode: "off".
You do not need to download anything by hand. On first start, the application
downloads the pre-converted OpenVINO model into
models/openvino/<model>/<weight_format>. Later starts reuse it.
Speculative Decoding (DFlash)#
Speculative decoding makes generation faster. A small draft model proposes several tokens, and the main model checks them in a single pass. DFlash is available for Qwen3.5-9B (INT4 and INT8) and Qwen3.6-35B-A3B (INT4).
Settings#
speculative:
mode: "on" # "off" | auto | "on"
method: dflash
draft_source: null
draft_model: null
draft_revision: null
num_assistant_tokens: 7 # null = default for the model
Setting |
Value |
Meaning |
|---|---|---|
|
|
Plain generation, no accelerator. Same as omitting the |
|
Use DFlash when it can be prepared and loaded. Otherwise, fall back to plain generation and report the reason in |
|
|
DFlash is required. If it cannot run, startup fails with an error that explains why. |
|
|
|
The accelerator to use. |
|
|
Use the tested DFlash draft model. It is downloaded and prepared automatically. |
|
|
Number of tokens drafted per step. |
Put quotes around "off" and "on". Without quotes, YAML reads them as
true/false.
Choosing num_assistant_tokens#
Start with the value in Recommended Configuration. A larger value drafts more tokens per step. It helps when most drafts are accepted, for example in code, math or structured output. It wastes work when many drafts are rejected. To tune it, change the value by 1 or 2, restart, and compare the speed on your own lessons.
What changes while DFlash is on#
Greedy output only. A request with
temperatureabove 0, or with sampling penalties, is rejected with HTTP 400. Smart Classroom’s own features already usetemperature: 0. This only affects your own clients on port 8000.Images still work. Requests with images use the same pipeline.
Slightly different wording. The output can differ in places from plain generation, with no consistent effect on accuracy.
Longer first start. The first start with DFlash downloads and prepares the draft model, which takes a few minutes. Later starts reuse it.
Check that DFlash is active#
Open http://127.0.0.1:8000/health after startup. Look at
hub.text_gen.serving.speculative:
"speculative": {
"requested_mode": "on",
"requested_method": "dflash",
"effective_method": "dflash",
"draft": "z-lab/Qwen3.5-9B-DFlash@5fc3b3d47476",
"num_assistant_tokens": 7,
"fallback_reason": null
}
"effective_method": "dflash" means DFlash is running. With mode: auto,
"effective_method": "plain" means the application fell back to plain
generation; fallback_reason says why.
Turn DFlash off#
Set mode: "off" and restart. The prepared files stay on disk for later use.
Model Files and Offline Use#
The application prepares the configured model when the backend starts. The paths
below are relative to %LOCALAPPDATA%\SmartClassroom.
Files |
Location |
|---|---|
Main model |
|
DFlash draft model |
|
An interrupted download resumes on the next start.
To run offline, start once with network access, or copy the prepared
models/folder (includingmodels/hf/) to the offline machine.
Tool Calling on Port 8000#
POST http://127.0.0.1:8000/v1/chat/completions accepts OpenAI-style function
tools on all models listed above. The server returns the tool calls; it never
runs them. Your client runs the tools and sends the results back.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
tools = [{"type": "function", "function": {
"name": "get_weather",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}},
"required": ["city"]}}}]
messages = [{"role": "user", "content": "Weather in Paris?"}]
first = client.chat.completions.create(model="text_gen", messages=messages, tools=tools)
call = first.choices[0].message.tool_calls[0] # finish_reason == "tool_calls"
messages += [first.choices[0].message,
{"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 21}'}]
answer = client.chat.completions.create(model="text_gen", messages=messages, tools=tools)
tool_choiceaccepts"none","auto"(default),"required", or a named function.Set
parallel_tool_calls: falseto get at most one call per turn.Add
"enable_thinking": false(inextra_body) to skip reasoning on Qwen3.5, Qwen3.6 and Qwen3.8 for faster calls. Reasoning, when on, is returned inmessage.reasoning_content.$refandpatternin tool schemas, and images given by URL or file path, are rejected with HTTP 400. Send images asdata:image/URLs.A call that does not match the tool’s schema returns HTTP 502 (
tool_call_validation_failed).
Design details are in vlm-llm-serving-design.md.