model - config.json
{
"architectures": [
"Gemma3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"attn_logit_softcapping": null,
"bos_token_id": 2,
"cache_implementation": "hybrid",
"eos_token_id": [
1,
106
],
"final_logit_softcapping": null,
"head_dim": 256,
"hidden_activation": "gelu_pytorch_tanh",
"hidden_size": 2560,
"initializer_range": 0.02,
"intermediate_size": 10240,
"max_position_embeddings": 32768,
"model_type": "gemma3",
"num_attention_heads": 8,
"num_hidden_layers": 34,
"num_key_value_heads": 4,
"pad_token_id": 0,
"query_pre_attn_scalar": 256,
"rms_norm_eps": 1e-06,
"rope_local_base_freq": 10000,
"rope_scaling": {
"factor": 1.0,
"type": "linear"
},
"rope_theta": 1000000,
"sliding_window_attention_size": 512,
"sliding_window_pattern": 6,
"global_local_attention_pattern": [
"SLIDING_LOCAL",
"SLIDING_LOCAL",
"SLIDING_LOCAL",
"SLIDING_LOCAL",
"SLIDING_LOCAL",
"GLOBAL"
],
"torch_dtype": "bfloat16",
"transformers_version": "4.50.0.dev0",
"use_cache": true,
"vocab_size": 262208,
"use_res_clamp": false
}
0_inference_float.sh can run well.
post_training_quantize$ bash 0_inference_float.sh
2026-07-30 16:58:08,749 [mtk_llm_sdk:INFO] inference_float.py:212: Please check if all arguments are correct:
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:213: Config file: ../models/gemma3_4b-it/config.json
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:216: Prompt inputs file: ./input_prompt.jsonl
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:217: Preformatter json: ./gemma_preformatter.json
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:218: Repetition penalty: 1.0
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:219: Simulated input token length: 128
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:220: Simulated cache size: 1024
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:221: Maximum output tokens: 1024
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:222: BOS mode: see
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:223: Is Rotated: False
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:232: Data type: float32
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:233: Save inference output to jsonl: None
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:234: Save inference prompt with preformatter:False
2026-07-30 16:58:08,750 [mtk_llm_sdk:INFO] inference_float.py:243: mtk_llm_sdk version: 3.4.3
2026-07-30 16:58:08,761 [mtk_llm_sdk:INFO] configuration_common.py:219: gemma3 config:
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:220: Hidden size: 2560
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:221: Intermediate size: 10240
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:222: Num layers: 34
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:223: Num attention heads: 8
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:224: Head dim: 256
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:225: Num KV heads: 4
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:226: Max pos emb: 32768
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:227: Rope theta: 1000000
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:231: Original max pos emb None
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:232: Rope scaling config: {'factor': 1.0, 'type': 'linear'}
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:233: Norm type: RMSNorm
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:234: Norm epsilon: 1e-06
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:235: BOS token id: 2
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:236: EOS token id: [1, 106]
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:237: PAD token id: 0
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:238: UNK token id: 0
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:239: Vocab size: 262208
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:240: Use stable embedding: False
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:241: Tie word embeddings: False
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:242: Use QK norm: True
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:243: Ring buffer: True
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:244: Use BiTA: False
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:248: Mask Scaling Factors: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:250: SWA window size: 512
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:274: Extra Input: sink_rope: False
2026-07-30 16:58:08,762 [mtk_llm_sdk:INFO] configuration_common.py:276: Extra Output: attn_logits: False
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_common.py:276: Extra Output: attn_weights: False
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_common.py:277: No RoPE interval: None
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_common.py:278: No RoPE layers: None
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_gemma.py:100: sliding_window_attention_sizes: 512
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_gemma.py:101: global_local_attention_pattern: ['SLIDING_LOCAL', 'SLIDING_LOCAL', 'SLIDING_LOCAL', 'SLIDING_LOCAL', 'SLIDING_LOCAL', 'GLOBAL']
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_gemma.py:102: rope_local_base_freq: 10000
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_gemma.py:103: use_vision: False
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_gemma.py:104: use_res_clamp: False
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_hook.py:66: Hook type: tokenizer_func_hook, hook name: default
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_hook.py:66: Hook type: get_embeds, hook name: text_only
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_hook.py:105: audio_token_ids: [-1]
2026-07-30 16:58:08,763 [mtk_llm_sdk:INFO] configuration_hook.py:106: image_token_ids: [-200]
2026-07-30 16:58:10,218 [mtk_llm_sdk:INFO] float_pipeline.py:754: Loading weights from:
2026-07-30 16:58:10,218 [mtk_llm_sdk:INFO] float_pipeline.py:758: ../models/gemma3_4b-it/model-00001-of-00002.safetensors
2026-07-30 16:58:16,420 [mtk_llm_sdk:INFO] float_pipeline.py:758: ../models/gemma3_4b-it/model-00002-of-00002.safetensors
2026-07-30 16:58:21,733 [mtk_llm_sdk:INFO] float_pipeline.py:260: Instantiating decoder layers...
2026-07-30 16:58:49,534 [mtk_llm_sdk:INFO] float_pipeline.py:142: Main device: cuda:0
2026-07-30 16:58:49,535 [mtk_llm_sdk:INFO] default.py:71: Input text (with gemma_preformatter preformatter):
<start_of_turn>user
请你介绍一下自己.<end_of_turn>
<start_of_turn>model
2026-07-30 16:58:49,544 [mtk_llm_sdk:INFO] base_pipeline.py:1328: Input Prompt Length: 15
2026-07-30 16:58:54,986 [mtk_llm_sdk:INFO] base_pipeline.py:1621: Response:
您好！我是 Gemma，一个由 Google DeepMind 训练的大型语言模型。我是一个开放权重的模型，这意味着我可以被广泛地使用。
我的主要功能是接收文本和图像作为输入，并生成文本作为输出。我可以尝试回答您的问题、提供信息、进行创意写作，甚至可以尝试理解和生成图像描述。
请随时向我提问或提出要求！
<end_of_turn>
2026-07-30 16:58:54,986 [mtk_llm_sdk:INFO] inference_float.py:321: ----------------------------------------------------------------------------------------------------
2026-07-30 16:58:55,886 [mtk_llm_sdk:INFO] memory_profiler.py:129: RAM-Peak = 12.83GB/540.64GB / VRAM-Peak = 22.12GB/103.08GB / Elapsed time: 47s
4_inference_quantized.sh got bad result after all quant steps.
post_training_quantize$ bash 4_inference_quantized.sh
2026-07-30 16:47:46,073 [mtk_llm_sdk:INFO] inference_quantized.py:349: Please check if all arguments are correct:
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:352: Prompt quantized model folder: quantized_models/gemma3_4b-it_gemma3_4b_instruct_precision_Overall_hessian/llm/llm_128t1024c
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:353: Generative quantized model folder: quantized_models/gemma3_4b-it_gemma3_4b_instruct_precision_Overall_hessian/llm/llm_1t1024c
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:354: Prompt inputs file: ./input_prompt.jsonl
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:357: Preformatter json: ./gemma_preformatter.json
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:358: Repetition penalty: 1.0
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:359: Prompt fixed shape input token length:128
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:360: Fixed shape cache size: 1024
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:361: Number of chunks: 1
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:362: Separate final FC: False
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:363: Maximum output tokens: 128
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:364: BOS mode: see
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:373: Use Single BMM Attention Graph: False
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:374: Save inference output to jsonl: None
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:375: Save inference prompt with preformatter:False
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:384: Inference Omni model: False
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:388: mtk_converter version: 9.8.0
2026-07-30 16:47:46,074 [mtk_llm_sdk:INFO] inference_quantized.py:391: mtk_llm_sdk version: 3.4.3
2026-07-30 16:47:47,556 [mtk_llm_sdk:INFO] quantized_pipeline.py:212: Loading generative model(s):
2026-07-30 16:47:47,556 [mtk_llm_sdk:INFO] quantized_model_utils.py:1737: Loading quantized_models/gemma3_4b-it_gemma3_4b_instruct_precision_Overall_hessian/llm/llm_1t1024c/gemma3_4b-it_gemma3_4b_instruct_precision_Overall_hessian_1t1024c_0.tflite
2026-07-30 16:49:03,433 [mtk_llm_sdk:INFO] quantized_pipeline.py:234: Loading prompt model(s):
2026-07-30 16:49:03,433 [mtk_llm_sdk:INFO] quantized_model_utils.py:1737: Loading quantized_models/gemma3_4b-it_gemma3_4b_instruct_precision_Overall_hessian/llm/llm_128t1024c/gemma3_4b-it_gemma3_4b_instruct_precision_Overall_hessian_128t1024c_0.tflite
2026-07-30 16:50:18,031 [mtk_llm_sdk:INFO] utils.py:1027: Loading embedding weights from `quantized_models/gemma3_4b-it_gemma3_4b_instruct_precision_Overall_hessian/embedding_int16.bin`.
2026-07-30 16:50:21,645 [mtk_llm_sdk:INFO] default.py:71: Input text (with gemma_preformatter preformatter):
<start_of_turn>user
请你介绍一下自己.<end_of_turn>
<start_of_turn>model
2026-07-30 16:50:21,653 [mtk_llm_sdk:INFO] base_pipeline.py:1328: Input Prompt Length: 15
2026-07-30 16:53:10,969 [mtk_llm_sdk:INFO] base_pipeline.py:1621: Response:
称 yourselves<pad> ? 嗎 ? ? ? ?<pad> ?<pad> ? ? ?
<pad> ? ?<pad> ? ? ? ?<pad> ? ?
我是 Gemma，一个大型的，开放式的，由 Google developed’s A. I. I. I<pad> fords.
<pad> for<pad> for<pad> for for<pad><pad> for<pad> for for for<pad> for for for for for<pad> for for<pad> for for for for for for for for<pad> for for for for for for for for for for for for<pad> for
我是个大型的，开放式的<pad> for 开放式的，由 Google developed’s A.<pad> for
2026-07-30 16:53:10,970 [mtk_llm_sdk:INFO] inference_quantized.py:503: ----------------------------------------------------------------------------------------------------
2026-07-30 16:53:11,676 [mtk_llm_sdk:INFO] memory_profiler.py:129: RAM-Peak = 15.51GB/540.64GB / VRAM-Peak = 17.31GB/103.08GB / Elapsed time: 5:25
my env:
mtk_converter 9.8.0
mtk_llm_sdk 3.4.3
mtk-quantization 9.0.1
Thanks.