If you are diving into Large Language Model (LLM) deployment on MediaTek NPUs, configuring the YAML file can feel like deciphering a foreign language. To make this accessible for embedded, hardware, and Linux kernel developers, here is the ultimate “Datasheet” for the MTK NPU deployment YAML.
We will break down these parameters by mapping them directly to the embedded hardware and low-level Linux concepts you already know.
Crucial Note on MTK’s
mainProgram: Before diving into the parameters, you must know a strict limitation of the current MediaTek environment. The underlying C++ code of the MTKmainexecutable is hardcoded to exclusively read this YAML configuration file. It no longer accepts external text file paths for prompts or configurations. Everything the engine needs to initialize must be defined within this YAML structure.
Here is the breakdown of the five core subsystems.
1. Hardware Topology & Bus Protocol Layer (Architecture & Types)
Operating Principle: strictly forbidden to modify randomly.
These parameters are “hard-soldered” by the algorithm team when compiling the .dla weight files. They act as the chip’s physical pinout and bus width definitions. Getting one parameter wrong will result in a direct Segmentation Fault or screen full of gibberish.
-
hiddenSize,numHead,numLayer,headDim: The physical topology of the neural network. This is equivalent to how many circuit layers or parallel processing units are inside the chip. You must configure these strictly according to the model’s release specs. -
modelInputType,modelOutputType,cacheType(e.g., INT16, INT8): The quantization precision of the data stream. Think of this as the sampling resolution of an ADC/DAC. The NPU memory requires strict byte alignment here. -
splitMask: The interrupt masking mechanism. This dictates whether Attention masks are processed jointly or split, depending on the lower-level driver’s requirements.
2. Memory Management & Throughput System (Memory & Throughput)
Operating Principle: Your primary tuning and optimization zone.
This section dictates whether your board will run out of memory (OOM) and how fast it processes data. It is the direct equivalent of allocating heap space in an RTOS and configuring DMA block sizes.
-
cacheSize: The Context Cache (KV Cache). This dictates how many conversation turns the model can remember. Setting it to 2048 means the model has a buffer of 2048 tokens. -
promptTokenBatchSize(e.g., 128): The DMA transfer block size during the prefill (time-to-first-token) phase. A larger value allows the NPU to swallow more words in parallel, speeding up the first-word response, but it causes massive instantaneous pressure on your VRAM bandwidth. -
sharedWeightsPaths: The ultimate memory-saving hack. This acts exactly like a Linux shared dynamic link library (.so). It allows the prefill model and the decoding (generation) model to share the foundational weights in physical memory, instantly freeing up hundreds of megabytes of RAM. -
dlaPaths(Format 2 Pseudo-dynamic Cache): This acts likemallocandreallocin C. You can configure different cache models (e.g., 512c, 1024c). As the conversation lengthens, the system will automatically “hot-swap” to the larger capacity model, squeezing every last megabyte out of the hardware.
3. Runtime Generation & Decoding Engine
Operating Principle: Currently sealed off by the C++ source code.
These parameters control the model’s “creativity” and logic. However, in the default MediaTek main.cpp testing tool you are compiling, these parameters are effectively bypassed. The source code currently hardcodes Argmax (greedy decoding—always picking the highest probability word). If you write your own C++ logic later, you will need these:
-
temperature: Injected “white noise.” Tuning it down (0.1) makes the model highly deterministic and rigid (great for coding). Tuning it up (1.0) introduces entropy, making the model more creative (great for poetry). -
topK,topP: Candidate filtering limiters. These act as band-pass filters to strip away extreme low-probability garbage outputs, preventing the model from hallucinating. -
repetitionPenalty: Equivalent to integral anti-windup in a PID controller. It artificially suppresses the probability of words it has already generated, preventing the system from getting stuck in an infinite loop.
4. Frame Format & Character Encoding Layer (Tokenizer & Protocol)
Operating Principle: Strict protocol alignment required.
The NPU only understands hex/binary numbers, not ASCII or UTF-8 characters. This section is identical to setting the baud rate, frame headers, and frame footers in UART serial communication.
-
bosId,eosId,stopToken: The Start Bit and Stop Bit. For example, when the NPU outputs theeosId(like the ID for<|im_end|>), the C++ program knows the data frame is complete and breaks out of the generationwhileloop. -
tokenizerRegex&tokenizerPath: The path to the local “hex-to-character” dictionary. This ensures the raw numerical IDs spat out by the NPU are correctly decoded back into human-readable text.
5. Advanced Commercial Engineering Hacks
Operating Principle: Hidden toggles to overclock product performance and UX.
-
cacheEvictionMode (LocalSnapKv): A Ring Buffer (FIFO) strategy. When enabled, if the conversation exceeds the hardware limit, the system automatically drops the oldest, least important middle tokens while preserving the system prompt and the newest messages. This achieves “infinite conversation without OOM.” -
cachePaths&startTokenIndex: The hibernation snapshot feature. If your system requires a fixed 1,000-word “system persona” on every boot, you can pre-calculate the KV states on a PC and save it as a.binfile. On boot, the board maps the.bindirectly into the NPU VRAM. This skips the calculation phase entirely, enabling a “zero-latency wake-up” for voice assistants. -
loraWeightsPaths: Hot-swappable modules (LoRA). You leave the main motherboard (base model) untouched, but dynamically mount a “coding” daughterboard when generating C code, or a “translation” daughterboard when translating text. -
numMedusaHeads: Branch Prediction. If the algorithm provides Medusa speculative decoding weights, the model attempts to guess the next 3-4 words in one clock cycle. If the prediction hits, they are accepted instantly, effectively doubling the generative token/s speed. -
rotEmbBase&ntkScale: Clock scaling and range extrapolation (RoPE scaling). When you want a small model to forcibly process text far beyond its designed context window, adjusting these is like stretching its internal “encoding ruler,” artificially expanding its field of view.