#Qwen3.6-35B-A3B-NVFP4 — DGX Spark

A vLLM deployment for Unsloth Qwen3.6-35B-A3B-NVFP4 on NVIDIA DGX Spark (GB10) — mixed FP8 dense + NVFP4 MoE, FlashInfer B12X linear GEMM (with soft fallback for non-NVFP4 layers), FP8 KV cache, and MTP speculative decoding.

---

#Key Features

---

#Benchmark

Decode benchmark on DGX Spark (GB10) with this stack: unsloth/Qwen3.6-35B-A3B-NVFP4, port 8888, 500 completion tokens, concurrency 1 / 2 / 3 / 4 / 6 / 8 (~1m 1s wall time).

• Aggregate — total decode tok/s across all concurrent streams
• Stream — average tok/s per stream

At concurrency 1, decode is ~95 tok/s with ~103 ms TTFT. Aggregate scales to ~317 tok/s at 8 concurrent streams.

---

#Prerequisites

---

#Quick Start

#1. Clone

git clone https://github.com/MiaAI-Lab/Unsloth-Qwen3.6-35b-NVFP4-DGX-Spark.git
cd Unsloth-Qwen3.6-35b-NVFP4-DGX-Spark

#2. (Optional) Hugging Face token

export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

#3. Start the server

./start.sh

start.sh will:

1. Check for Docker and curl
2. Use a local image if present; otherwise docker pull ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x:latest (image is resolved before model download, since download can fall back to docker run with this image)
3. Ensure the model is available in the local HF cache (download if needed)
4. Launch with --gpus all, host network/IPC, large shared memory, CUTE_DSL_ARCH=sm_121a
5. Stream logs and wait until http://127.0.0.1:8888/v1/models is ready

Example output:

Using local image: ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x:latest
# (or: Pulling image: ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x:latest)
Model unsloth/Qwen3.6-35B-A3B-NVFP4 is already cached in ...
Starting vLLM container for unsloth/Qwen3.6-35B-A3B-NVFP4
Image: ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x:latest
Listening on 0.0.0.0:8888
...
vLLM is ready!
OpenAI base URL: http://0.0.0.0:8888/v1

First load can take several minutes (model + kernel warmup). Later restarts are much faster if weights are cached under .cache/huggingface/.

#4. Test

curl -s http://127.0.0.1:8888/v1/models | jq

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unsloth/Qwen3.6-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "What is 19 × 23?"}],
    "temperature": 0,
    "max_tokens": 100
  }' | jq

#5. Stop

./stop.sh

---

#Docker image

#Why a custom image?

Stock vLLM treats --linear-backend as a hard filter: if a layer type has no kernel for that backend, EngineCore exits with:

ValueError: --linear-backend=flashinfer_b12x was requested but no
'flashinfer_b12x' kernel exists for this layer type.

Unsloth Qwen3.6 NVFP4 is mixed (FP8 dense linears + NVFP4 experts/linears). B12X is the right path for NVFP4 GEMM, but FP8 (and other) linears need their own kernels.

This image patches linear selection so that:

Base image already reports b12x gemm True and b12x moe True on GB10; the patch only changes forced-backend behavior.

#Pull

docker pull ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x:latest

The package is published on GHCR under MiaAI-Lab. If a future tag is private or rate-limited, log in:

echo "$GITHUB_TOKEN" | docker login ghcr.io -u YOUR_GITHUB_USER --password-stdin

#Rebuild on Spark (optional)

Build on the DGX Spark (aarch64), not on an x86 PC:

./docker/build.sh
# optional overrides:
# BASE_IMAGE=ghcr.io/timothystewart6/vllm-gb10:latest \
# TAG=ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x:latest \
# ./docker/build.sh

Verify GPU + B12X + patch:

docker run --rm --gpus all --entrypoint python3 \
  ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x:latest \
  /usr/local/bin/verify_b12x.py

Expected:

cap (12, 1)
b12x gemm True | b12x moe True
linear soft-fallback patch True
OK: image ready for --linear-backend flashinfer_b12x on mixed NVFP4

#Image sources (docker/)

---

#Configuration

Options live in start.sh.

#vLLM flags (as launched)

#Runtime environment (container)

#Container runtime

---

#Linear B12X vs MoE B12X

For tok/s on Spark, MoE backend choice usually matters more than linear B12X. Linear B12X still helps NVFP4 GEMM layers when available.

---

#Recommended client configuration (coding)

OpenAI-compatible API on port 8888.

#Provider

{
  "baseUrl": "http://localhost:8888/v1",
  "api": "openai-completions",
  "apiKey": "dummy",
  "compat": {
    "supportsDeveloperRole": false,
    "supportsReasoningEffort": false,
    "maxTokensField": "max_tokens"
  }
}

#Model entry

{
  "id": "unsloth/Qwen3.6-35B-A3B-NVFP4",
  "name": "Unsloth Qwen3.6 35B A3B NVFP4",
  "reasoning": true,
  "input": ["text", "image"],
  "contextWindow": 262144,
  "maxTokens": 32000,
  "params": {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0,
    "presence_penalty": 0,
    "repetition_penalty": 1,
    "chat_template_kwargs": {
      "enable_thinking": true,
      "preserve_thinking": true
    }
  }
}

The model field in requests must match the served model id (defaults to the Hugging Face model id).

Sampling: For precise coding (thinking mode), temperature=0.6, top_p=0.95, top_k=20 is a good default. For general thinking tasks, Qwen often suggests higher temperature (e.g. 1.0). See the Unsloth Qwen3.6 guide.

---

#Performance notes

• ~80% GPU memory (--gpu-memory-utilization 0.80); model weights on the order of ~22 GiB.
• 24 concurrent sequences and 32 768 batched tokens tuned for long context + batching.
• MTP (2 tokens) improves decode; acceptance rate depends on workload.
• FlashInfer B12X linear path for NVFP4 GEMM on SM121.
• FP8 KV cache for memory efficiency.
• Decode / TTFT: see Benchmark (~95 tok/s and ~103 ms TTFT at concurrency 1; ~317 tok/s aggregate at ×8). Numbers are workload-dependent.

---

#Troubleshooting

---

#License

• Model weights: see Unsloth model card
• This codebase: MIT License
• Base image: vllm-gb10 (MIT) + upstream vLLM / FlashInfer licenses

---

#Resources

• Package on GHCR
• vllm-gb10 (base Spark image)
• Unsloth NVFP4 guide
• Unsloth Qwen3.6
• vLLM docs
• FlashInfer
• Model on Hugging Face