Using 2 GPUs on vLLM 0.27.1 (tensor parallelism = 2)
sudo docker run -t -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=XXXXXXXXXXXXXX" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 2 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --max-model-len 262144 --max_num_seqs 128 --gpu-memory-utilization 0.96
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|-------:|-----------------:|--------------:|--------------:|---------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 | 6670.65 ± 103.98 | | 387.83 ± 4.72 | 307.24 ± 4.72 | 387.83 ± 4.72 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 | 145.04 ± 4.56 | 152.33 ± 0.47 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c4) | 5668.69 ± 32.41 | 2785.43 ± 1925.05 | | | 1095.21 ± 417.30 | 1015.99 ± 417.30 | 1095.21 ± 417.30 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c4) | 333.05 ± 5.04 | 96.41 ± 7.26 | 420.33 ± 2.87 | 105.17 ± 0.99 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c8) | 5830.13 ± 29.08 | 1766.29 ± 1657.68 | | | 1859.85 ± 792.08 | 1781.12 ± 792.08 | 1859.85 ± 792.08 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c8) | 450.09 ± 10.60 | 70.37 ± 7.33 | 658.67 ± 3.77 | 82.38 ± 0.48 | | | |
Using 3 GPUs on vLLM 0.27.1 (pipeline parallelism = 3)
sudo docker run -t -d --name vllm --restart unless-stopped --runtime nvidia --gpus '"device=0,1,2"' -v ~/.cache/huggingface:/root/.cache/huggingface --env "HF_TOKEN=XXXXXXXXXXXXXXXXX" -e HOST='0.0.0.0' -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --trust-remote-code --enable-auto-tool-choice --tensor-parallel-size 1 --pipeline-parallel-size 3 --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --max-model-len 262144 --gpu-memory-utilization 0.96
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|-------:|-----------------:|---------------:|---------------:|---------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 | 9425.73 ± 549.01 | | 346.51 ± 12.38 | 218.03 ± 12.38 | 346.51 ± 12.38 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 | 97.73 ± 5.37 | 110.00 ± 11.52 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|----------------:|----------------:|----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c4) | 6566.67 ± 36.49 | 3748.84 ± 2944.48 | | | 941.75 ± 361.16 | 821.21 ± 361.16 | 941.75 ± 361.16 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c4) | 254.66 ± 6.30 | 71.56 ± 3.69 | 308.00 ± 0.00 | 77.17 ± 0.55 | | | |
llama-benchy --base-url http://localhost:8000/v1 --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-----------------------------------|------------:|----------------:|------------------:|--------------:|-----------------:|-----------------:|-----------------:|-----------------:|
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | pp2048 (c8) | 6732.56 ± 65.44 | 2240.45 ± 2322.53 | | | 1611.31 ± 680.65 | 1491.84 ± 680.65 | 1611.31 ± 680.65 |
| ornith-ai/Ornith-1.5-35B-A3B-NVFP4 | tg512 (c8) | 353.03 ± 9.06 | 53.08 ± 3.75 | 485.33 ± 3.77 | 60.67 ± 0.47 | | | |
Using 3 GPUs on Ollama 0.32.15
llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 1 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------|-------:|-----------------:|--------------:|---------------:|---------------:|----------------:|
| ornith-1.5:35b | pp2048 | 2302.64 ± 212.34 | | 920.73 ± 72.15 | 826.61 ± 72.15 | 920.73 ± 72.15 |
| ornith-1.5:35b | tg512 | 116.40 ± 2.91 | 117.00 ± 2.83 | | | |
llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 4 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------|------------:|---------------:|----------------:|--------------:|-----------------:|------------------:|------------------:|------------------:|
| ornith-1.5:35b | pp2048 (c4) | 462.18 ± 21.53 | 730.62 ± 931.06 | | | 8726.79 ± 5759.04 | 8624.00 ± 5759.04 | 8726.79 ± 5759.04 |
| ornith-1.5:35b | tg512 (c4) | 100.62 ± 0.69 | 116.26 ± 2.88 | 119.67 ± 0.47 | 116.67 ± 2.95 | | |
llama-benchy --base-url http://localhost:11434/v1 --model ornith-1.5:35b --depth 0 --pp 2048 --tg 512 --concurrency 8 --runs 3 --latency-mode generation --no-cache --save-total-throughput-timeseries
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------|------------:|--------------:|----------------:|--------------:|-----------------:|--------------------:|--------------------:|--------------------:|
| ornith-1.5:35b | pp2048 (c8) | 396.01 ± 8.87 | 390.72 ± 724.14 | | | 19186.55 ± 11898.78 | 19084.22 ± 11898.78 | 19186.55 ± 11898.78 |
| ornith-1.5:35b | tg512 (c8) | 97.21 ± 1.60 | 114.69 ± 4.75 | 119.67 ± 0.47 | 115.12 ± 4.75 | | | |
No comments:
Post a Comment