Is anyone using vLLM on a Genio720 running Ubuntu Linux? When running the vLLM benchmarking code, I noticed that the throughput is extremely low (compared to llama.cpp). What could be causing this issue, and how can it be resolved?
Here is my testing method:
For vLLM:
$ taskset -c 6,7 vllm bench throughput
–model meta-llama/Llama-3.2-1B-Instruct
–dataset-name random
–num-prompts 50
–random-input-len 128
–random-output-len 128
–max-model-len 2048
Throughput: 0.01 requests/s, 1.75 total tokens/s, 0.87 output tokens/s
For comparison, here is the performance of llama.cpp:
$./llama-bench -m ../models/Llama-3.2-1B-Instruct-f16.gguf -p 128 -n 128 -t 2 -r 5 -o csv > ./genio720_llamacpp_bench.csv
| Inference Engine |
Model |
Size |
Threads |
Test |
Tokens/sec (t/s) |
| llama.cpp |
llama 1B F16 |
2.30 GiB |
2 |
pp128 (Prefill) |
51.03 ± 0.04 |
| llama.cpp |
llama 1B F16 |
2.30 GiB |
2 |
tg128 (Decode) |
7.31 ± 0.00 |
Hello @Jim_Huang , Thanks for sharing the benchmark results. The performance difference is certainly interesting, and I’d be happy to investigate this with you.
At the moment, MediaTek’s NPU is not supported by either vLLM or llama.cpp, so these benchmarks are expected to be running on the CPU rather than the Genio 720 NPU.
We also have not yet formally validated vLLM or llama-bench performance on the Genio 720, so we don’t currently have an internal benchmark to compare against these results.
One possible factor is the ARM64 CPU optimization in the two projects. According to the vLLM documentation, CPU/ARM64 support is currently more limited compared with its primary GPU-focused platforms, and the available CPU optimization paths can also depend on the specific ARM architecture and instruction set.
In contrast, llama.cpp has a significant amount of ARM64-specific optimization, including NEON-based SIMD kernels and other ARM CPU optimizations. These could potentially make a substantial difference on the Genio 720, although we would need some additional measurements to confirm this.