Ornith-1.5-35B-A3B-NVFP4 benchmark on RTX 5060 Ti 16GB

Using 2 GPUs on vLLM 0.27.1 (tensor parallelism = 2)

sudo docker run -t -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=XXXXXXXXXXXXXX" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 2 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --max-model-len 262144 --max_num_seqs 128 --gpu-memory-utilization 0.96


llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                              |   test |              t/s |      peak t/s |     ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------------|-------:|-----------------:|--------------:|--------------:|---------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 | 6670.65 ± 103.98 |               | 387.83 ± 4.72 |  307.24 ± 4.72 |   387.83 ± 4.72 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 |  tg512 |    145.04 ± 4.56 | 152.33 ± 0.47 |               |                |                 |


llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                              |        test |     t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c4) | 5668.69 ± 32.41 | 2785.43 ± 1925.05 |               |                  | 1095.21 ± 417.30 | 1015.99 ± 417.30 | 1095.21 ± 417.30 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 |  tg512 (c4) |   333.05 ± 5.04 |      96.41 ± 7.26 | 420.33 ± 2.87 |    105.17 ± 0.99 |                  |                  |                  |


llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                              |        test |     t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c8) | 5830.13 ± 29.08 | 1766.29 ± 1657.68 |               |                  | 1859.85 ± 792.08 | 1781.12 ± 792.08 | 1859.85 ± 792.08 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 |  tg512 (c8) |  450.09 ± 10.60 |      70.37 ± 7.33 | 658.67 ± 3.77 |     82.38 ± 0.48 |                  |                  |                  |

Using 3 GPUs on vLLM 0.27.1 (pipeline parallelism = 3)

sudo docker run -t -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=0,1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=XXXXXXXXXXXXXXXXX" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 1 --pipeline-parallel-size 3 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --max-model-len 262144 --gpu-memory-utilization 0.96

llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                              |   test |              t/s |       peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------------|-------:|-----------------:|---------------:|---------------:|---------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 | 9425.73 ± 549.01 |                | 346.51 ± 12.38 | 218.03 ± 12.38 |  346.51 ± 12.38 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 |  tg512 |     97.73 ± 5.37 | 110.00 ± 11.52 |                |                |                 |


llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                              |        test |     t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |       ttfr (ms) |    est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|----------------:|----------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c4) | 6566.67 ± 36.49 | 3748.84 ± 2944.48 |               |                  | 941.75 ± 361.16 | 821.21 ± 361.16 | 941.75 ± 361.16 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 |  tg512 (c4) |   254.66 ± 6.30 |      71.56 ± 3.69 | 308.00 ± 0.00 |     77.17 ± 0.55 |                 |                 |                 |


llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                              |        test |     t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c8) | 6732.56 ± 65.44 | 2240.45 ± 2322.53 |               |                  | 1611.31 ± 680.65 | 1491.84 ± 680.65 | 1611.31 ± 680.65 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 |  tg512 (c8) |   353.03 ± 9.06 |      53.08 ± 3.75 | 485.33 ± 3.77 |     60.67 ± 0.47 |                  |                  |                  |

Using 3 GPUs on Ollama 0.32.15

llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model          |   test |              t/s |      peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:---------------|-------:|-----------------:|--------------:|---------------:|---------------:|----------------:|
| ornith-1.5:35b | pp2048 | 2302.64 ± 212.34 |               | 920.73 ± 72.15 | 826.61 ± 72.15 |  920.73 ± 72.15 |
| ornith-1.5:35b |  tg512 |    116.40 ± 2.91 | 117.00 ± 2.83 |                |                |                 |


llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model          |        test |    t/s (total) |       t/s (req) |      peak t/s |   peak t/s (req) |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:---------------|------------:|---------------:|----------------:|--------------:|-----------------:|------------------:|------------------:|------------------:|
| ornith-1.5:35b | pp2048 (c4) | 462.18 ± 21.53 | 730.62 ± 931.06 |               |                  | 8726.79 ± 5759.04 | 8624.00 ± 5759.04 | 8726.79 ± 5759.04 |
| ornith-1.5:35b |  tg512 (c4) |  100.62 ± 0.69 |   116.26 ± 2.88 | 119.67 ± 0.47 |    116.67 ± 2.95 |                   |                   |      


llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model          |        test |   t/s (total) |       t/s (req) |      peak t/s |   peak t/s (req) |           ttfr (ms) |        est_ppt (ms) |       e2e_ttft (ms) |
|:---------------|------------:|--------------:|----------------:|--------------:|-----------------:|--------------------:|--------------------:|--------------------:|
| ornith-1.5:35b | pp2048 (c8) | 396.01 ± 8.87 | 390.72 ± 724.14 |               |                  | 19186.55 ± 11898.78 | 19084.22 ± 11898.78 | 19186.55 ± 11898.78 |
| ornith-1.5:35b |  tg512 (c8) |  97.21 ± 1.60 |   114.69 ± 4.75 | 119.67 ± 0.47 |    115.12 ± 4.75 |                     |                     |                     |

Local LLM on three nVidia RTX 5060 Ti 16GB using nvidia/Qwen3.6-35B-A3B-NVFP4

Finally managed to get the nvidia/Qwen3.6-35B-A3B-NVFP4 to run across three RTX 5060 Ti 16GB with 256k context using vLLM's pipeline parallelism:

sudo docker run -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=0,1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=xxxxxxxx" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:nightly --model nvidia/Qwen3.6-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 1 --pipeline-parallel-size 3 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --mm-encoder-tp-mode data --enable-prefix-caching --gpu-memory-utilization 0.96

utilization from nvtop:

 Device 0 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 59.55 MiB/s TX: 25.50 MiB/s
 GPU 2797MHz MEM 13801MH TEMP  53°C  FAN  32%   POW  52 / 180 W
 GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.023Gi/15.929Gi]

 Device 1 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 1.785 GiB/s TX: 320.9 MiB/s
 GPU 2790MHz MEM 13801MH TEMP  54°C  FAN  35%   POW  56 / 180 W
 GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.005Gi/15.929Gi]

 Device 2 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 2.148 GiB/s TX: 338.1 MiB/s
 GPU 2790MHz MEM 13801MH TEMP  54°C  FAN  32%   POW  60 / 180 W
 GPU[||||||||||||||||||||||||||       71%] MEM[||||||||||||||||||||14.539Gi/15.929Gi]
   ┌──────────────────────────────────────────┐   ┌──────────────────────────────────────────┐   ┌──────────────────────────────────────────┐
100│GPU0 %──────┐ ┌─────┐ ┌───────────────────│100│GPU1 %──────┐ ┌─────┐ ┌───────────────────│100│GPU2 %                                    │
   │GPU0 mem%───┼─┼─────┴─┴───────────────────│   │GPU1 mem%───┼─┼─────┼─┼───────────────────│   │GPU2 mem%───────────┬─┬───────────────────│
 75│            │ │                           │ 75│            │ │     └─┘                   │ 75│  ┌─────────┐ ┌─┐   │ └───────┐   ┌─┐     │
   │            └─┘                           │   │            └─┘                           │   │──┘         │ │ └───┘         │   │ │ ┌───│
   │                                          │   │                                          │   │            └─┘               └───┘ └─┘   │
 50│                                          │ 50│                                          │ 50│                                          │
   │                                          │   │                                          │   │                                          │
 25│                                          │ 25│                                          │ 25│                                          │
   │                                          │   │                                          │   │                                          │
  0│                                          │  0│                                          │  0│                                          │
   └21s──────15s────────10s───────5s────────0s┘   └21s──────15s────────10s───────5s────────0s┘   └21s──────15s────────10s───────5s────────0s┘
    PID USER DEV     TYPE  GPU        GPU MEM    CPU  HOST MEM Command
 933979 root   2  Compute  71%  14456MiB  89%    82%   5565MiB VLLM::Worker_PP2
 933977 root   0  Compute  99%  13928MiB  85%    72%   5925MiB VLLM::Worker_PP0
 933978 root   1  Compute  99%  13910MiB  85%    75%   5455MiB VLLM::Worker_PP1

and the benchmark result:

llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |   test |               t/s |      peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------|-------:|------------------:|--------------:|---------------:|---------------:|----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 | 7621.64 ± 1816.91 |               | 409.04 ± 57.99 | 282.69 ± 57.99 |  409.04 ± 57.99 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 |     115.19 ± 8.86 | 118.33 ± 8.01 |                |                |                 |


llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |        test |      t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |       ttfr (ms) |    est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------------------|------------:|-----------------:|------------------:|--------------:|-----------------:|----------------:|----------------:|----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c4) | 5814.28 ± 168.19 | 3834.94 ± 2748.07 |               |                  | 910.26 ± 387.12 | 785.86 ± 387.12 | 910.26 ± 387.12 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 (c4) |    257.64 ± 1.27 |      69.65 ± 3.35 | 306.67 ± 1.89 |     76.67 ± 0.47 |                 |                 |                 |


llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries

| model                        |        test |     t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-----------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c8) | 6026.16 ± 38.26 | 2415.53 ± 2407.68 |               |                  | 1573.88 ± 755.91 | 1446.76 ± 755.91 | 1573.88 ± 755.91 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 |  tg512 (c8) |   364.16 ± 1.92 |      51.75 ± 3.59 | 491.33 ± 2.49 |     61.42 ± 0.49 |                  |                  |                  |

Useful MCP servers for Claude Code

For web content scraping

claude mcp add playwright npx @playwright/mcp@latest