How to Compile Llama 3.2 1B FP LM Head to DLA on Genio 720? Output Dimension 128256 Exceeds MDLA Limit

NeuroPilot NeuroPilot - GenAI
,
1

Hi MediaTek team,

I am testing Llama 3.2 1B Instruct on Genio 720 using the GAI Deployment Toolkit and NeuroPilot SDK.

Environment:

  • Platform: MediaTek Genio 720
  • GAI Deployment Toolkit:
    GAI-Deployment-Toolkit-v2.0.6_llama3.2-1b-3b-v0.1
  • NeuroPilot SDK:
    neuropilot-sdk-basic-8.0.11-build20260211
  • Target:
    MDLA 5.3 / EDMA 3.6
  • Model:
    Llama 3.2 1B Instruct

I have already successfully converted and compiled the 8W16A version to DLA.

I am now testing the floating-point path using:

mtk_ptq_llm
converter
../Llama3.2-1B-Instruct/config.json
-p FP

The FP conversion succeeds.

The generated model directory contains:

  • FP TFLite files
  • embedding_fp16.bin

Based on the TFLite file sizes, the Transformer / LM Head weights appear to remain FP32, while the embedding LUT is exported as FP16.

I then applied Shape Fixer using:

mtk_fix_llm_shape
tflite/Llama3.2-1B-Instruct_FP
128t1024c 1t1024c
-n 4 4 4 2 2

This succeeds and creates five Prompt chunks and five Generative chunks:

[4, 4, 4, 2, 2]

Host-side TFLite inference also works correctly.

Both Prompt and Decode models can be loaded successfully, and text generation produces normal results.

For DLA compilation, I found that the original LLM compile script triggers:

FATAL:
LLM_BMASB_with_CachePattern
only support quantized data type (no DQ)

Therefore, I tested a minimal FP compile command:

LD_LIBRARY_PATH=“$NEURON_SDK/host/lib”
“$NEURON_SDK/host/bin/ncc-tflite”
–arch=mdla5.3,edma3.6
–relax-fp32
<model.tflite>

With this command, the Transformer chunks can successfully be compiled into DLA.

For example:

Llama3.2-1B-Instruct_FP_4layer_128t1024c_0.tflite
→ successfully generates
Llama3.2-1B-Instruct_FP_4layer_128t1024c_0.dla

The Decode chunk also compiles successfully.

However, the final chunk containing the LM Head fails.

The compiler reports:

OP[111]: FULLY_CONNECTED
├ MDLA: Dimension should be <= 65535.
Operand: 1 got <128256 x 2048>.
├ MDLA: Dimension should be <= 65535.
Result: 0 got <128 x 128256>.
├ EDMA: unsupported operation

ERROR: Cannot find an execution plan because of unsupported operations

The problem appears to come from the Llama 3.2 vocabulary size:

Vocab size = 128256

LM Head:
2048 → 128256

Since 128256 exceeds the MDLA single-dimension limit of 65535, the last chunk cannot be compiled.

I also tested:

--fc-to-conv

The FC is converted, but the output channel is still 128256:

Operand:
<128256 x 1 x 1 x 2048>

Result:
<1 x 8 x 16 x 128256>

so compilation still fails.

I also tried:

--split-16a4w-conv-oc

but it does not split the FP graph.

This appears reasonable because this option seems to be specifically designed for the 16A4W quantized path.

For comparison, the official W4A16 flow successfully generates the complete DLA model and uses:

embedding_int16.bin

The W4A16 compiler flow also provides 16A4W-specific output-channel splitting.

In contrast, the FP path produces:

embedding_fp16.bin

and I have not found an equivalent generic FP LM Head / output-channel split option.

I also checked the ncc-tflite binary and found internal passes such as:

  • SplitFCPass
  • ConvertFcToConv2DPass
  • ConvertFloat32ToFloat16Pass
  • ConvertFloat16ToFloat32Pass

and LLM configurations such as:

  • llm_prompt_accuracy
  • llm_prompt_aggressive
  • llm_gen_accuracy
  • llm_gen_aggressive

However, neither --help nor --help-hidden exposes a generic FP SplitFCPass or LM Head output-channel split option.

My current status is therefore:

FP conversion PASS
Shape Fixer PASS
Prompt / Decode TFLite inference PASS
FP Transformer chunks → DLA PASS
Final LM Head chunk → DLA FAIL

Could you please confirm:

  1. Is FP Llama 3.2 1B officially supported for full DLA compilation on Genio 720?

  2. Is there a compiler option or recommended recipe for splitting the FP LM Head when vocab_size = 128256 exceeds the MDLA 65535 dimension limit?

  3. Is SplitFCPass available through any public ncc-tflite option or compiler preset?

  4. Is the current behavior a limitation of NeuroPilot SDK 8.0.11, or has this been addressed in a newer compiler version?

  5. If full FP LLM DLA is not supported, is the recommended approach to use 16W16A / W8A16 / W4A16 for LLM deployment on Genio 720?

Thank you.

3

Hi Crchiu,

Thanks for reaching out!

Currently, on Genio 720 using the NeuroPilot SDK and GAI Deployment Toolkit, LLM models need to be compiled and deployed to the DLA through our official quantization workflow.

Best,
Jun

4

Hi Jun,

Thanks for your reply.

I would like to clarify our question, because our current test is already based on the official GAI Deployment Toolkit and NeuroPilot SDK.

The key point is that the GAI Deployment Toolkit itself provides an FP conversion path:

mtk_ptq_llm converter <config.json> -p FP

Using this official FP flow, we can successfully complete:

  • FP model conversion

  • Shape Fixer

  • Prompt / Decode TFLite generation

  • Host-side TFLite inference

  • Correct text generation

  • FP Transformer chunk compilation to DLA

The only failure occurs in the final chunk containing the LM Head.

For Llama 3.2 1B, the LM Head output dimension is:

2048 → 128256

and ncc-tflite reports:

MDLA: Dimension should be <= 65535

because the vocabulary dimension 128256 exceeds the MDLA limit.

Therefore, our question is not whether we should use the official NeuroPilot / GAI workflow. We are already using it.

What we would like to confirm is the intended support scope of the FP option provided by the GAI Deployment Toolkit.

Could you please clarify the following points?

  1. Does -p FP officially support end-to-end DLA deployment for Llama 3.2 1B on Genio 720, or is the FP option intended only for TFLite / host-side validation?

  2. If FP DLA deployment is supported, what is the recommended compiler option or flow for handling the LM Head output dimension of 128256, which exceeds the MDLA 65535 dimension limit?

  3. We can see that the quantized W4A16 flow has specific output-channel splitting support, but we could not find an equivalent mechanism for the FP graph. Is there any public option, compiler preset, or supported recipe for splitting the FP LM Head?

  4. If FP LLM models are not supported for full DLA compilation, could you please confirm that the current expected deployment path for Genio 720 is to use a quantized configuration such as W4A16 / W8A16 / 16W16A?

In other words, we would like to understand whether the following behavior is expected:

FP conversion → PASS
FP TFLite inference → PASS
FP Transformer DLA compilation → PASS
FP LM Head DLA compilation → FAIL due to vocab dimension > 65535

If this is an expected limitation of the current SDK/compiler, please also let us know whether it has been addressed in a newer NeuroPilot SDK version.

Thanks for your help.

5

Hi crchiu,

Regarding the LM Head compilation failure where the vocabulary size (128256) exceeds the MDLA dimension limit (65535), this can be resolved by enabling gno=LTS in your compiler configuration.

With gno=LTS enabled, FC layers with output dimensions greater than 65535 are supported, provided the dimension is divisible by 16.

Please try recompiling with gno=LTS enabled to see if it resolves the issue.

Best,
Jun