Local LLM on three nVidia RTX 5060 Ti 16GB using nvidia/Qwen3.6-35B-A3B-NVFP4

Finally managed to get the nvidia/Qwen3.6-35B-A3B-NVFP4 to run across three RTX 5060 Ti 16GB with 256k context using vLLM's pipeline parallelism:

sudo docker run -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=0,1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=xxxxxxxx" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:nightly --model nvidia/Qwen3.6-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 1 --pipeline-parallel-size 3 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --mm-encoder-tp-mode data --enable-prefix-caching --gpu-memory-utilization 0.96

utilization from nvtop:

 Device 0 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 59.55 MiB/s TX: 25.50 MiB/s
 GPU 2797MHz MEM 13801MH TEMP  53°C  FAN  32%   POW  52 / 180 W
 GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.023Gi/15.929Gi]

 Device 1 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 1.785 GiB/s TX: 320.9 MiB/s
 GPU 2790MHz MEM 13801MH TEMP  54°C  FAN  35%   POW  56 / 180 W
 GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.005Gi/15.929Gi]

 Device 2 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 2.148 GiB/s TX: 338.1 MiB/s
 GPU 2790MHz MEM 13801MH TEMP  54°C  FAN  32%   POW  60 / 180 W
 GPU[||||||||||||||||||||||||||       71%] MEM[||||||||||||||||||||14.539Gi/15.929Gi]
   ┌──────────────────────────────────────────┐   ┌──────────────────────────────────────────┐   ┌──────────────────────────────────────────┐
100│GPU0 %──────┐ ┌─────┐ ┌───────────────────│100│GPU1 %──────┐ ┌─────┐ ┌───────────────────│100│GPU2 %                                    │
   │GPU0 mem%───┼─┼─────┴─┴───────────────────│   │GPU1 mem%───┼─┼─────┼─┼───────────────────│   │GPU2 mem%───────────┬─┬───────────────────│
 75│            │ │                           │ 75│            │ │     └─┘                   │ 75│  ┌─────────┐ ┌─┐   │ └───────┐   ┌─┐     │
   │            └─┘                           │   │            └─┘                           │   │──┘         │ │ └───┘         │   │ │ ┌───│
   │                                          │   │                                          │   │            └─┘               └───┘ └─┘   │
 50│                                          │ 50│                                          │ 50│                                          │
   │                                          │   │                                          │   │                                          │
 25│                                          │ 25│                                          │ 25│                                          │
   │                                          │   │                                          │   │                                          │
  0│                                          │  0│                                          │  0│                                          │
   └21s──────15s────────10s───────5s────────0s┘   └21s──────15s────────10s───────5s────────0s┘   └21s──────15s────────10s───────5s────────0s┘
    PID USER DEV     TYPE  GPU        GPU MEM    CPU  HOST MEM Command
 933979 root   2  Compute  71%  14456MiB  89%    82%   5565MiB VLLM::Worker_PP2
 933977 root   0  Compute  99%  13928MiB  85%    72%   5925MiB VLLM::Worker_PP0
 933978 root   1  Compute  99%  13910MiB  85%    75%   5455MiB VLLM::Worker_PP1

and the benchmark result:

llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |   test |               t/s |      peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------|-------:|------------------:|--------------:|---------------:|---------------:|----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 | 7621.64 ± 1816.91 |               | 409.04 ± 57.99 | 282.69 ± 57.99 |  409.04 ± 57.99 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 |     115.19 ± 8.86 | 118.33 ± 8.01 |                |                |                 |


llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |        test |      t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |       ttfr (ms) |    est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------|------------:|-----------------:|------------------:|--------------:|-----------------:|----------------:|----------------:|----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c4) | 5814.28 ± 168.19 | 3834.94 ± 2748.07 |               |                  | 910.26 ± 387.12 | 785.86 ± 387.12 | 910.26 ± 387.12 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 (c4) |    257.64 ± 1.27 |      69.65 ± 3.35 | 306.67 ± 1.89 |     76.67 ± 0.47 |                 |                 |                 |


llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |        test |     t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-----------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c8) | 6026.16 ± 38.26 | 2415.53 ± 2407.68 |               |                  | 1573.88 ± 755.91 | 1446.76 ± 755.91 | 1573.88 ± 755.91 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 (c8) |   364.16 ± 1.92 |      51.75 ± 3.59 | 491.33 ± 2.49 |     61.42 ± 0.49 |                  |                  |                  |