Qwen3.8-27B benchmark on 3x RTX 5060 Ti 16GB on Ollama 0.32.15

I managed to get the model running with 256k context on my 3x RTX 5060 Ti 16GB setup.  Following Parameters were used:

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q4_0"
Environment="OLLAMA_SCHED_SPREAD=1"
Environment="OLLAMA_CONTEXT_LENGTH=262144"

And the results using llama-benchy:

llama-benchy --base-url http://localhost:11434/v1 --model qwen3.8:27b --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model       |   test |           t/s |     peak t/s |       ttfr (ms) |    est_ppt (ms) |   e2e_ttft (ms) |
|:------------|-------:|--------------:|-------------:|----------------:|----------------:|----------------:|
| qwen3.8:27b | pp2048 | 645.39 ± 5.82 |              | 2979.82 ± 95.73 | 2738.63 ± 95.73 | 2979.82 ± 95.73 |
| qwen3.8:27b |  tg512 |  40.68 ± 1.89 | 41.33 ± 2.05 |                 |                 |                 |


llama-benchy --base-url http://localhost:11434/v1 --model qwen3.8:27b --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model       |        test |   t/s (total) |       t/s (req) |     peak t/s |   peak t/s (req) |           ttfr (ms) |        est_ppt (ms) |       e2e_ttft (ms) |
|:------------|------------:|--------------:|----------------:|-------------:|-----------------:|--------------------:|--------------------:|--------------------:|
| qwen3.8:27b | pp2048 (c4) | 139.64 ± 2.14 | 209.82 ± 259.08 |              |                  | 28291.80 ± 18803.48 | 28062.33 ± 18803.48 | 28291.80 ± 18803.48 |
| qwen3.8:27b |  tg512 (c4) |  32.37 ± 1.13 |    38.26 ± 3.17 | 42.67 ± 2.62 |     38.75 ± 3.11 |                     |                     |                     |


llama-benchy --base-url http://localhost:11434/v1 --model qwen3.8:27b --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model       |        test |   t/s (total) |       t/s (req) |     peak t/s |   peak t/s (req) |           ttfr (ms) |        est_ppt (ms) |
|:------------|------------:|--------------:|----------------:|-------------:|-----------------:|--------------------:|--------------------:|-------
| qwen3.8:27b | pp2048 (c8) | 125.29 ± 3.90 | 118.09 ± 211.66 |              |                  | 61020.08 ± 38195.67 | 60794.71 ± 38195.67 | 61020.
| qwen3.8:27b |  tg512 (c8) |  31.23 ± 1.19 |    37.53 ± 2.75 | 41.33 ± 1.25 |     37.96 ± 2.70 |                     |                     |

Next goal is to get the model running in vLLM and see how it compares.

Unable to boot from USB thumb drive on Supermicro motherboard (X10DRC-T4+)

There's a weird bug in Supermicro motherboards, where you cannot boot from (a known good bootable) USB thumb drive.  To get around this, make sure that the Video EPROM setting is set to Legacy (not UEFI)

Local LLM on three nVidia RTX 5060 Ti 16GB using nvidia/Qwen3.6-35B-A3B-NVFP4

Finally managed to get the nvidia/Qwen3.6-35B-A3B-NVFP4 to run across three RTX 5060 Ti 16GB with 256k context using vLLM's pipeline parallelism:

sudo docker run -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=0,1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=xxxxxxxx" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:nightly --model nvidia/Qwen3.6-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 1 --pipeline-parallel-size 3 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --mm-encoder-tp-mode data --enable-prefix-caching --gpu-memory-utilization 0.96

utilization from nvtop:

 Device 0 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 59.55 MiB/s TX: 25.50 MiB/s
 GPU 2797MHz MEM 13801MH TEMP  53°C  FAN  32%   POW  52 / 180 W
 GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.023Gi/15.929Gi]

 Device 1 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 1.785 GiB/s TX: 320.9 MiB/s
 GPU 2790MHz MEM 13801MH TEMP  54°C  FAN  35%   POW  56 / 180 W
 GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.005Gi/15.929Gi]

 Device 2 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 2.148 GiB/s TX: 338.1 MiB/s
 GPU 2790MHz MEM 13801MH TEMP  54°C  FAN  32%   POW  60 / 180 W
 GPU[||||||||||||||||||||||||||       71%] MEM[||||||||||||||||||||14.539Gi/15.929Gi]
   ┌──────────────────────────────────────────┐   ┌──────────────────────────────────────────┐   ┌──────────────────────────────────────────┐
100│GPU0 %──────┐ ┌─────┐ ┌───────────────────│100│GPU1 %──────┐ ┌─────┐ ┌───────────────────│100│GPU2 %                                    │
   │GPU0 mem%───┼─┼─────┴─┴───────────────────│   │GPU1 mem%───┼─┼─────┼─┼───────────────────│   │GPU2 mem%───────────┬─┬───────────────────│
 75│            │ │                           │ 75│            │ │     └─┘                   │ 75│  ┌─────────┐ ┌─┐   │ └───────┐   ┌─┐     │
   │            └─┘                           │   │            └─┘                           │   │──┘         │ │ └───┘         │   │ │ ┌───│
   │                                          │   │                                          │   │            └─┘               └───┘ └─┘   │
 50│                                          │ 50│                                          │ 50│                                          │
   │                                          │   │                                          │   │                                          │
 25│                                          │ 25│                                          │ 25│                                          │
   │                                          │   │                                          │   │                                          │
  0│                                          │  0│                                          │  0│                                          │
   └21s──────15s────────10s───────5s────────0s┘   └21s──────15s────────10s───────5s────────0s┘   └21s──────15s────────10s───────5s────────0s┘
    PID USER DEV     TYPE  GPU        GPU MEM    CPU  HOST MEM Command
 933979 root   2  Compute  71%  14456MiB  89%    82%   5565MiB VLLM::Worker_PP2
 933977 root   0  Compute  99%  13928MiB  85%    72%   5925MiB VLLM::Worker_PP0
 933978 root   1  Compute  99%  13910MiB  85%    75%   5455MiB VLLM::Worker_PP1

and the benchmark result:

llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |   test |               t/s |      peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------|-------:|------------------:|--------------:|---------------:|---------------:|----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 | 7621.64 ± 1816.91 |               | 409.04 ± 57.99 | 282.69 ± 57.99 |  409.04 ± 57.99 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 |     115.19 ± 8.86 | 118.33 ± 8.01 |                |                |                 |


llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |        test |      t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |       ttfr (ms) |    est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------|------------:|-----------------:|------------------:|--------------:|-----------------:|----------------:|----------------:|----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c4) | 5814.28 ± 168.19 | 3834.94 ± 2748.07 |               |                  | 910.26 ± 387.12 | 785.86 ± 387.12 | 910.26 ± 387.12 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 (c4) |    257.64 ± 1.27 |      69.65 ± 3.35 | 306.67 ± 1.89 |     76.67 ± 0.47 |                 |                 |                 |


llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |        test |     t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-----------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c8) | 6026.16 ± 38.26 | 2415.53 ± 2407.68 |               |                  | 1573.88 ± 755.91 | 1446.76 ± 755.91 | 1573.88 ± 755.91 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 (c8) |   364.16 ± 1.92 |      51.75 ± 3.59 | 491.33 ± 2.49 |     61.42 ± 0.49 |                  |                  |                  |

To disable Windows 11 RDP client from displaying warnings every time you're using a known-good RDP file, run this as an Administrator:

reg add "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services\Client" /v RedirectionWarningDialogVersion /t REG_DWORD /d 1 /f