[Solution] Fix OOM (Killed) at the final layer when quantizing Qwen3-4B via mtk_ptq_llm

NeuroPilot NeuroPilot - GenAI
1

Issue

When running mtk_ptq_llm to quantize Qwen3-4B, the process is abruptly Killed (OOM) during the “Weight Quantization Parameter Optimization” phase, typically at the final layer.

Cause

Using the Hessian algorithm (-w 'hessian') consumes massive amounts of RAM. Processing the entire model at once can easily exhaust system memory, triggering the Linux OOM killer.

Solution

Use the --partial flag to process the problematic layer (e.g., Layer 36) individually. This drastically reduces peak memory usage.

Here is the updated script (02_mtk_ptq_llm.sh):

Bash

mtk_ptq_llm \
    converter \
    ../models/Qwen3-4B/config_splitMask.json \
    -d ./calibration_datasets/Qwen3-4B_splitMask \
    -p ../models/Qwen3-4B/qwen3_4B_precision_DX3.json \
    --np_compat 8 \
    -m 'Overall' \
    -w 'hessian' \
    --extra_converter_options ../models/Qwen3-4B/wgt_opt_cum_layer_error.json \
    --partial l36 \
    -o ./dynamic_tflite_output_layer36

Key Changes:

  • --partial l36: Isolates Layer 36 for independent conversion and optimization, bypassing the massive memory overhead of loading all layers.

  • -o ./dynamic_tflite_output_layer36: Redirects the output to a new folder to avoid overwriting your full-model output directory.

Tip: If your machine has less than 64GB of RAM, adding Swap space (64GB+) is highly recommended when doing Hessian optimization for LLMs.