sudo dmidecode -t memory
Saturday, September 12, 2026
Tuesday, August 25, 2026
Ornith-1.5-35B-A3B-NVFP4 benchmark on RTX 5060 Ti 16GB
Using 2 GPUs on vLLM 0.27.1 (tensor parallelism = 2)
sudo docker run -t -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=XXXXXXXXXXXXXX" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 2 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --max-model-len 262144 --max_num_seqs 128 --gpu-memory-utilization 0.96
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|-------:|-----------------:|--------------:|--------------:|---------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 | 6670.65 ± 103.98 | | 387.83 ± 4.72 | 307.24 ± 4.72 | 387.83 ± 4.72 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 | 145.04 ± 4.56 | 152.33 ± 0.47 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c4) | 5668.69 ± 32.41 | 2785.43 ± 1925.05 | | | 1095.21 ± 417.30 | 1015.99 ± 417.30 | 1095.21 ± 417.30 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c4) | 333.05 ± 5.04 | 96.41 ± 7.26 | 420.33 ± 2.87 | 105.17 ± 0.99 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c8) | 5830.13 ± 29.08 | 1766.29 ± 1657.68 | | | 1859.85 ± 792.08 | 1781.12 ± 792.08 | 1859.85 ± 792.08 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c8) | 450.09 ± 10.60 | 70.37 ± 7.33 | 658.67 ± 3.77 | 82.38 ± 0.48 | | | |
Using 3 GPUs on vLLM 0.27.1 (pipeline parallelism = 3)
sudo docker run -t -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=0,1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=XXXXXXXXXXXXXXXXX" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 1 --pipeline-parallel-size 3 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --max-model-len 262144 --gpu-memory-utilization 0.96
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|-------:|-----------------:|---------------:|---------------:|---------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 | 9425.73 ± 549.01 | | 346.51 ± 12.38 | 218.03 ± 12.38 | 346.51 ± 12.38 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 | 97.73 ± 5.37 | 110.00 ± 11.52 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|----------------:|----------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c4) | 6566.67 ± 36.49 | 3748.84 ± 2944.48 | | | 941.75 ± 361.16 | 821.21 ± 361.16 | 941.75 ± 361.16 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c4) | 254.66 ± 6.30 | 71.56 ± 3.69 | 308.00 ± 0.00 | 77.17 ± 0.55 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c8) | 6732.56 ± 65.44 | 2240.45 ± 2322.53 | | | 1611.31 ± 680.65 | 1491.84 ± 680.65 | 1611.31 ± 680.65 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c8) | 353.03 ± 9.06 | 53.08 ± 3.75 | 485.33 ± 3.77 | 60.67 ± 0.47 | | | |
Using 3 GPUs on Ollama 0.32.15
llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------|-------:|-----------------:|--------------:|---------------:|---------------:|----------------:|
| ornith-1.5:35b | pp2048 | 2302.64 ± 212.34 | | 920.73 ± 72.15 | 826.61 ± 72.15 | 920.73 ± 72.15 |
| ornith-1.5:35b | tg512 | 116.40 ± 2.91 | 117.00 ± 2.83 | | | |
llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------|------------:|---------------:|----------------:|--------------:|-----------------:|------------------:|------------------:|------------------:|
| ornith-1.5:35b | pp2048 (c4) | 462.18 ± 21.53 | 730.62 ± 931.06 | | | 8726.79 ± 5759.04 | 8624.00 ± 5759.04 | 8726.79 ± 5759.04 |
| ornith-1.5:35b | tg512 (c4) | 100.62 ± 0.69 | 116.26 ± 2.88 | 119.67 ± 0.47 | 116.67 ± 2.95 | | |
llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------|------------:|--------------:|----------------:|--------------:|-----------------:|--------------------:|--------------------:|--------------------:|
| ornith-1.5:35b | pp2048 (c8) | 396.01 ± 8.87 | 390.72 ± 724.14 | | | 19186.55 ± 11898.78 | 19084.22 ± 11898.78 | 19186.55 ± 11898.78 |
| ornith-1.5:35b | tg512 (c8) | 97.21 ± 1.60 | 114.69 ± 4.75 | 119.67 ± 0.47 | 115.12 ± 4.75 | | | |
Friday, August 21, 2026
Qwen3.8-27B benchmark on 3x RTX 5060 Ti 16GB
I managed to get the model running with 256k context on my 3x RTX 5060 Ti 16GB setup. Following Parameters were used on Ollama 0.32.15:
[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q4_0"
Environment="OLLAMA_SCHED_SPREAD=1"
Environment="OLLAMA_CONTEXT_LENGTH=262144"
And the results using llama-benchy:
llama-benchy --base-url http://localhost:11434/v1 --model qwen3.8:27b --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:------------|-------:|--------------:|-------------:|----------------:|----------------:|----------------:|
| qwen3.8:27b | pp2048 | 645.39 ± 5.82 | | 2979.82 ± 95.73 | 2738.63 ± 95.73 | 2979.82 ± 95.73 |
| qwen3.8:27b | tg512 | 40.68 ± 1.89 | 41.33 ± 2.05 | | | |
llama-benchy --base-url http://localhost:11434/v1 --model qwen3.8:27b --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:------------|------------:|--------------:|----------------:|-------------:|-----------------:|--------------------:|--------------------:|--------------------:|
| qwen3.8:27b | pp2048 (c4) | 139.64 ± 2.14 | 209.82 ± 259.08 | | | 28291.80 ± 18803.48 | 28062.33 ± 18803.48 | 28291.80 ± 18803.48 |
| qwen3.8:27b | tg512 (c4) | 32.37 ± 1.13 | 38.26 ± 3.17 | 42.67 ± 2.62 | 38.75 ± 3.11 | | | |
llama-benchy --base-url http://localhost:11434/v1 --model qwen3.8:27b --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) |
|:------------|------------:|--------------:|----------------:|-------------:|-----------------:|--------------------:|--------------------:|-------
| qwen3.8:27b | pp2048 (c8) | 125.29 ± 3.90 | 118.09 ± 211.66 | | | 61020.08 ± 38195.67 | 60794.71 ± 38195.67 | 61020.
| qwen3.8:27b | tg512 (c8) | 31.23 ± 1.19 | 37.53 ± 2.75 | 41.33 ± 1.25 | 37.96 ± 2.70 | | |
I also managed to get the NVFP4 model running vLLM 0.28.0 using 2x GPUs, albeit with much smaller context length:
sudo docker run -t -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=XXXXXXXXXXXX" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model unsloth/Qwen3.8-27B-NVFP4 --tensor-parallel-size 2 --gpu-memory-utilization 0.96 --max-model-len 64K --max_num_seqs 8
and the benchmark:
llama-benchy --base-url http://localhost:8000/v1 --model unsloth/Qwen3.8-27B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:--------------------------|-------:|----------------:|-------------:|----------------:|----------------:|----------------:|
| unsloth/Qwen3.8-27B-NVFP4 | pp2048 | 2004.90 ± 21.32 | | 1208.68 ± 11.06 | 1022.28 ± 11.06 | 1208.68 ± 11.06 |
| unsloth/Qwen3.8-27B-NVFP4 | tg512 | 33.79 ± 0.00 | 35.00 ± 0.00 | | | |Tuesday, August 18, 2026
Unable to boot from USB thumb drive on Supermicro motherboard (X10DRC-T4+)
There's a weird bug in Supermicro motherboards, where you cannot boot from (a known good bootable) USB thumb drive. To get around this, make sure that the Video EPROM setting is set to Legacy (not UEFI)
Thursday, August 13, 2026
Local LLM on three nVidia RTX 5060 Ti 16GB using nvidia/Qwen3.6-35B-A3B-NVFP4
Finally managed to get the nvidia/Qwen3.6-35B-A3B-NVFP4 to run across three RTX 5060 Ti 16GB with 256k context using vLLM's pipeline parallelism:
sudo docker run -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=0,1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=xxxxxxxx" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:nightly --model nvidia/Qwen3.6-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 1 --pipeline-parallel-size 3 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --mm-encoder-tp-mode data --enable-prefix-caching --gpu-memory-utilization 0.96
utilization from nvtop:
Device 0 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 59.55 MiB/s TX: 25.50 MiB/s
GPU 2797MHz MEM 13801MH TEMP 53°C FAN 32% POW 52 / 180 W
GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.023Gi/15.929Gi]
Device 1 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 1.785 GiB/s TX: 320.9 MiB/s
GPU 2790MHz MEM 13801MH TEMP 54°C FAN 35% POW 56 / 180 W
GPU[||||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||||14.005Gi/15.929Gi]
Device 2 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 3@ 8x RX: 2.148 GiB/s TX: 338.1 MiB/s
GPU 2790MHz MEM 13801MH TEMP 54°C FAN 32% POW 60 / 180 W
GPU[|||||||||||||||||||||||||| 71%] MEM[||||||||||||||||||||14.539Gi/15.929Gi]
┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────┐
100│GPU0 %──────┐ ┌─────┐ ┌───────────────────│100│GPU1 %──────┐ ┌─────┐ ┌───────────────────│100│GPU2 % │
│GPU0 mem%───┼─┼─────┴─┴───────────────────│ │GPU1 mem%───┼─┼─────┼─┼───────────────────│ │GPU2 mem%───────────┬─┬───────────────────│
75│ │ │ │ 75│ │ │ └─┘ │ 75│ ┌─────────┐ ┌─┐ │ └───────┐ ┌─┐ │
│ └─┘ │ │ └─┘ │ │──┘ │ │ └───┘ │ │ │ ┌───│
│ │ │ │ │ └─┘ └───┘ └─┘ │
50│ │ 50│ │ 50│ │
│ │ │ │ │ │
25│ │ 25│ │ 25│ │
│ │ │ │ │ │
0│ │ 0│ │ 0│ │
└21s──────15s────────10s───────5s────────0s┘ └21s──────15s────────10s───────5s────────0s┘ └21s──────15s────────10s───────5s────────0s┘
PID USER DEV TYPE GPU GPU MEM CPU HOST MEM Command
933979 root 2 Compute 71% 14456MiB 89% 82% 5565MiB VLLM::Worker_PP2
933977 root 0 Compute 99% 13928MiB 85% 72% 5925MiB VLLM::Worker_PP0
933978 root 1 Compute 99% 13910MiB 85% 75% 5455MiB VLLM::Worker_PP1
and the benchmark result:
llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-----------------------------|-------:|------------------:|--------------:|---------------:|---------------:|----------------:| | nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 | 7621.64 ± 1816.91 | | 409.04 ± 57.99 | 282.69 ± 57.99 | 409.04 ± 57.99 | | nvidia/Qwen3.6-35B-A3B-NVFP4 | tg512 | 115.19 ± 8.86 | 118.33 ± 8.01 | | | | llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-----------------------------|------------:|-----------------:|------------------:|--------------:|-----------------:|----------------:|----------------:|----------------:| | nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c4) | 5814.28 ± 168.19 | 3834.94 ± 2748.07 | | | 910.26 ± 387.12 | 785.86 ± 387.12 | 910.26 ± 387.12 | | nvidia/Qwen3.6-35B-A3B-NVFP4 | tg512 (c4) | 257.64 ± 1.27 | 69.65 ± 3.35 | 306.67 ± 1.89 | 76.67 ± 0.47 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model nvidia/Qwen3.6-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-----------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:| | nvidia/Qwen3.6-35B-A3B-NVFP4 | pp2048 (c8) | 6026.16 ± 38.26 | 2415.53 ± 2407.68 | | | 1573.88 ± 755.91 | 1446.76 ± 755.91 | 1573.88 ± 755.91 | | nvidia/Qwen3.6-35B-A3B-NVFP4 | tg512 (c8) | 364.16 ± 1.92 | 51.75 ± 3.59 | 491.33 ± 2.49 | 61.42 ± 0.49 | | | |
Saturday, August 1, 2026
Tuesday, May 19, 2026
Create a cheap DIY CFexpress Type B storage using an M.2 NVMe 2230 SSD drive
I recently found a way to save significantly on CFexpress Type B storage and wanted to share the details with you. While looking for storage for my new camera, I realized that CFexpress is essentially a repackaged NVMe SSD, despite the high retail price.
I successfully built a 2TB CFexpress Type B card for $314, compared to the $1,200+ retail cost (e.g. OWC Atlas Ultra 2.0TB), using the following components:
- Adapter: RIITOP CFexpress B to NVMe 2230 SSD Adapter
I hope this DIY solution helps you get the storage you need at a much better price. Let me know if you have any questions!
I successfully built a 2TB CFexpress Type B card for $314, compared to the $1,200+ retail cost (e.g. OWC Atlas Ultra 2.0TB), using the following components:
- Adapter: RIITOP CFexpress B to NVMe 2230 SSD Adapter
For my Canon EOS R6 Mark III to recognize the drive, I initialized it using MBR partitioning (rather than GPT) and formatted it to exFAT. In benchmarks, the drive sustained over 1GB/sec read and 1.7GB/sec write speeds, which easily exceeds the VPG-400 requirement of 400MB/sec.
I hope this DIY solution helps you get the storage you need at a much better price. Let me know if you have any questions!
Subscribe to:
Posts (Atom)