Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+figures/benchmark.jpeg filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,17 @@
+MINIMAX COMMUNITY LICENSE
+Copyright (c) 2026 MiniMax
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software for non-commercial purposes, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or provide copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+1. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+2. If the Software (or any derivative works thereof) is used for any Commercial Use for your products or services: 
+  1. you shall prominently display “Built with MiniMax M3” on a related website, user interface, blogpost, about page or product documentation. 
+  2. you shall obtain a separate, prior written authorization from MiniMax by contacting api@minimax.io with the subject line “M3 licensing - authorization request”, if such products and services generate more than 20 million US dollars (or equivalent in other currencies) in yearly revenue; otherwise, you only need to send a one-time notice to api@minimax.io with the subject “M3 licensing — notice”.
+3. “Commercial Use” means any use of the Software or any derivative work thereof that is primarily intended for commercial advantage or monetary compensation, which includes, without limitation: (i) offering products or services to third parties for a fee, which utilize, incorporate, or rely on the Software or its derivatives, (ii) the commercial use of APIs provided by or for the Software or its derivatives, including to support or enable commercial products, services, or operations, whether in a cloud-based, hosted, or other similar environment, and (iii) the deployment or provision of the Software or its derivatives that have been subjected to post-training, fine-tuning, instruction-tuning, or any other form of modification, for any commercial purpose.
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+Appendix: Prohibited Uses 
+You agree you will not use, or allow others to use, the Software or any derivatives of the Software to:
+1. Generate or disseminate content prohibited by applicable laws or regulations.
+2. Assist with, engage in or otherwise support any military purpose.
+3. Exploit, harm, or attempt to exploit or harm minors.
+4. Generate or disseminate false or misleading information with the intent to cause harm.
+5. Promote discrimination, hate speech, or harmful behavior against individuals or groups based on race or ethnic origin, religion, disability, age, nationality and national origin, veteran status, sexual orientation, gender or gender identity, caste, immigration status, or any other characteristic that is associated with systemic discrimination or marginalization.

README.md

@@ -0,0 +1,78 @@
+---
+license: other
+license_name: minimax
+license_link: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/main/LICENSE
+base_model: MiniMaxAI/MiniMax-M3
+base_model_relation: quantized
+tags:
+  - nvfp4
+  - fp4
+  - modelopt
+  - minimax
+  - moe
+  - quantized
+---
+
+# MiniMax-M3-NVFP4
+
+**The first NVFP4 quantization of [MiniMaxAI/MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3)** — 428B-total / 23B-active MoE with MiniMax Sparse Attention (MSA), quantized 2026-06-12.
+
+- **~256 GB** on disk (vs 854 GB BF16, 444 GB MXFP8) → serves on **2× B300/GB300** with huge KV headroom, or fits tighter Blackwell pairs
+- Routed + shared **experts in NVFP4** (group 16, two-level scaling); attention, MSA indexer, router, embeddings, lm_head and the vision tower kept in **original BF16** (no quantization round-trip — copied verbatim from the source checkpoint)
+- Same recipe family as `nvidia/MiniMax-M2.7-NVFP4` (experts-only NVFP4), produced with **TensorRT Model Optimizer 0.44.0**
+
+## Quantization recipe
+
+| | |
+|---|---|
+| Method | PTQ, NVFP4 (FP4 weights+activations, FP8 per-16 block scales + FP32 global) |
+| Tool | nvidia-modelopt 0.44.0, transformers main (native `minimax_m3_vl`) |
+| Calibration | 512 samples × 2048 tokens: cnn_dailymail + nvidia/OpenCodeReasoning + nvidia/OpenMathReasoning |
+| Quantized | routed experts (w1/w2/w3) + shared experts, all 57 MoE layers |
+| Excluded | attention (incl. MSA indexer), router/gate, embeddings, lm_head, vision tower, projectors |
+| KV cache | not quantized (v1; MSA is young in engines — don't stack experiments) |
+
+Calibration deliberately uses longer sequences and reasoning traces than the modelopt defaults: NVFP4 activation scales are per-block *dynamic* at runtime, so calibration only pins the per-tensor global scales — reasoning-heavy data exposes the activation extremes a thinking model actually produces.
+
+## Evals
+
+Measured via lm-evaluation-harness against the official **MiniMax-M3-MXFP8** endpoint as baseline, same engine (SGLang), same sampling (temperature 1.0, top-p 0.95, model-card settings), thinking enabled. Generation caps: 16384 tokens (GPQA, MMLU), 8192 (GSM8K); at these caps truncation is negligible (<0.25% of samples).
+
+| Task | MXFP8 (official) | **NVFP4 (this repo)** |
+|---|---|---|
+| GSM8K (5-shot, strict) | 93.93 | 92.57 |
+| GPQA diamond (CoT zero-shot) | 76.26 | 69.70 |
+| MMLU (flan CoT few-shot, 25% sample) | 77.36 | 74.16 |
+
+## Serving (SGLang)
+
+Requires the MiniMax-M3 bring-up image and **one patch**: SGLang's NVFP4 cutlass MoE path does not yet forward M3's clamped-swiglu activation parameters (`swiglu_alpha=1.702`, `swiglu_limit=7.0`, `+1` beta — same family as GPT-OSS). Without the patch the model loads but generates garbage. The patched files ship in this repo under `sglang_patch/` (upstream PR pending; the second file makes the unsupported flashinfer-trtllm MoE backend fail fast instead of producing garbage).
+
+```bash
+docker run --runtime=nvidia --gpus '"device=0,1"' --ipc=host --shm-size 32g \
+  -v $MODEL_DIR:/model \
+  -v $MODEL_DIR/sglang_patch/modelopt_quant.py:/sgl-workspace/sglang/python/sglang/srt/layers/quantization/modelopt_quant.py:ro \
+  -v $MODEL_DIR/sglang_patch/flashinfer_trtllm.py:/sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/flashinfer_trtllm.py:ro \
+  -p 30014:30014 lmsysorg/sglang:dev-cu13-minimax-m3 \
+  sglang serve --model-path /model --tp 2 \
+    --quantization modelopt_fp4 \
+    --attention-backend fa4 --page-size 128 \
+    --moe-runner-backend flashinfer_cutlass \
+    --context-length 131072 --mem-fraction-static 0.90 \
+    --reasoning-parser auto --tool-call-parser auto \
+    --trust-remote-code --host 0.0.0.0 --port 30014
+```
+
+`--page-size 128` is mandatory (MSA indexing). Sampling: temperature 1.0, top-p 0.95, top-k 40.
+
+`--moe-runner-backend flashinfer_cutlass` is the only supported MoE backend: the flashinfer-trtllm FP4 kernels cannot run M3's parameterized clamped swiglu (they ignore `gemm1_alpha`/`gemm1_beta`; the patched code fails fast with a clear error instead of generating garbage).
+
+## Known limitations
+
+- Engine support is bleeding-edge: M3 itself has not shipped in stable SGLang/vLLM; this NVFP4 additionally needs the swiglu-parameter fix (upstream PR pending).
+- Vision tower is BF16 and untested under this serving path beyond loading; the eval table is text-only.
+- KV cache quantization intentionally omitted in v1.
+
+## Provenance
+
+Quantized from `MiniMaxAI/MiniMax-M3` (revision 3a41b31) on 8× NVIDIA B300. During bring-up, two upstream gaps were found and fixed: modelopt's fused-experts detector did not recognize M3's `_apply_gate` expert module (experts silently skipped — PR pending), and SGLang's NVFP4 cutlass MoE path dropped custom swiglu parameters (PR pending).

added_tokens.json

@@ -0,0 +1,63 @@
+{
+  "]!p~[": 200000,
+  "<fim_prefix>": 200001,
+  "<fim_middle>": 200002,
+  "<fim_suffix>": 200003,
+  "<fim_pad>": 200004,
+  "<reponame>": 200005,
+  "<filename>": 200006,
+  "<gh_stars>": 200007,
+  "<issue_start>": 200008,
+  "<issue_comment>": 200009,
+  "<issue_closed>": 200010,
+  "<jupyter_start>": 200011,
+  "<jupyter_text>": 200012,
+  "<jupyter_code>": 200013,
+  "<jupyter_output>": 200014,
+  "<empty_output>": 200015,
+  "<commit_before>": 200016,
+  "<commit_msg>": 200017,
+  "<commit_after>": 200018,
+  "]~b]": 200019,
+  "[e~[": 200020,
+  "]!d~[": 200021,
+  "<function_call>": 200022,
+  "<code_interpreter>": 200023,
+  "]<]speech[>[": 200024,
+  "]<]image[>[": 200025,
+  "]<]video[>[": 200026,
+  "]<]start of speech[>[": 200027,
+  "]<]end of speech[>[": 200028,
+  "]<]start of image[>[": 200029,
+  "]<]end of image[>[": 200030,
+  "]<]start of video[>[": 200031,
+  "]<]end of video[>[": 200032,
+  "]<]vision pad[>[": 200033,
+  "]~!b[": 200034,
+  "<jupyter_error>": 200035,
+  "<add_file>": 200036,
+  "<delete_file>": 200037,
+  "<rename_file>": 200038,
+  "<edit_file>": 200039,
+  "<commit_message>": 200040,
+  "<empty_source_file>": 200041,
+  "<repo_struct>": 200042,
+  "<code_context>": 200043,
+  "<file_content>": 200044,
+  "<source_files>": 200045,
+  "<pr_start>": 200046,
+  "<review_comment>": 200047,
+  "<filepath>": 200048,
+  "<file_sep>": 200049,
+  "<think>": 200050,
+  "</think>": 200051,
+  "<tool_call>": 200052,
+  "</tool_call>": 200053,
+  "]<]frame[>[": 200054,
+  "]<]start of frame[>[": 200055,
+  "]<]end of frame[>[": 200056,
+  "<|content_altered_placeholder|>": 200057,
+  "]<]minimax[>[": 200058,
+  "<mm:think>": 200059,
+  "</mm:think>": 200060
+}

chat_template.jinja

@@ -0,0 +1,247 @@
+{# ---------- special token variables ---------- #}
+{%- set ns_token               = ']<]minimax[>['                  -%}
+{%- set bod_token              = ']~!b['                          -%}
+{%- set bos_token              = ']~b]'                           -%}
+{%- set eos_token              = '[e~['                           -%}
+{%- set toolcall_begin_token   = ns_token ~ '<tool_call>'         -%}
+{%- set toolcall_end_token     = ns_token ~ '</tool_call>'        -%}
+{%- set think_begin_token      = '<mm:think>'                     -%}
+{%- set think_end_token        = '</mm:think>'                    -%}
+{%- set image_token            = ']<]image[>['                    -%}
+{%- set video_token            = ']<]video[>['                    -%}
+{#- Thinking mode: "enabled" / "disabled" / "adaptive" / not defined -#}
+{#- Recursive XML renderer for tool_call arguments ======================== -#}
+{#- None values are intentionally skipped in mapping iteration so that
+    `<key>null</key>` (which would round-trip to the literal string "null")
+    never appears in the rendered tool_call. The convention is: omit the
+    field entirely. The top-level `_args` loop applies the same rule.
+    The `val is none` branch below is a safety net only — upstream cleaning
+    (drop_none_in_tool_arguments) should ensure no None ever reaches here. -#}
+{%- macro to_xml(val, ns) -%}
+{%- if val is mapping -%}
+{%- for k, v in val.items() if v is not none -%}
+{{ ns }}<{{ k }}>{{ to_xml(v, ns) }}{{ ns }}</{{ k }}>
+{%- endfor -%}
+{%- elif val is iterable and val is not string -%}
+{%- for item in val -%}
+{{ ns }}<item>{{ to_xml(item, ns) }}{{ ns }}</item>
+{%- endfor -%}
+{%- elif val is none -%}
+{#- Should be unreachable when upstream cleaning is applied. -#}
+{%- elif val is boolean -%}
+{{ val | tojson }}
+{%- else -%}
+{{ val }}
+{%- endif -%}
+{%- endmacro -%}
+{#- Tool Rendering Functions ============================================== -#}
+{%- macro render_tool_namespace(namespace_name, tool_list) -%}
+{%- for tool in tool_list -%}
+<tool>{{ tool.function | tojson(ensure_ascii=False) }}</tool>
+{% endfor -%}
+{%- endmacro -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{ content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is mapping and item.type == 'image' -%}
+                {{- image_token }}
+            {%- elif item is mapping and item.type == 'video' -%}
+                {{- video_token}}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- elif content is none -%}
+        {{- '' }}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{#- System Message Construction ============================================ -#}
+{%- macro build_system_message(system_message) -%}
+    {%- if system_message and system_message.content -%}
+        {{- visible_text(system_message.content) }}
+    {%- else -%}
+        {{- 'Your model version is MiniMax-M3, developed by MiniMax. Knowledge cutoff: January 2026. Founded in early 2022, MiniMax is a global AI foundation model company committed to advancing the frontiers of AI towards AGI.' }}
+    {%- endif -%}
+
+    {#- Thinking mode instructions -#}
+    {{- '\n\n<thinking_instructions>\n' }}
+    {{- 'You have a thinking capability that allows you to reason step by step before responding. When thinking is enabled, wrap your reasoning in ' ~ think_begin_token ~ think_end_token ~ ' tags before your response. When thinking is disabled, begin your response directly after the ' ~ think_end_token ~ ' prefix. When thinking is adaptive, decide on your own whether to think for the current turn.\n' }}
+    {%- if thinking_mode is defined -%}
+        {%- if thinking_mode == "enabled" -%}
+            {{- 'Current thinking mode: enabled. You MUST think step by step before every response, including after receiving function/tool results.\n' }}
+        {%- elif thinking_mode == "disabled" -%}
+            {{- 'Current thinking mode: disabled. Do not output any thinking process.\n' }}
+        {%- elif thinking_mode == "adaptive" -%}
+            {{- 'Current thinking mode: adaptive. You are encouraged to think for complex decision-making, multi-step reasoning, or when analyzing function/tool results.\n' }}
+        {%- endif -%}
+    {%- else -%}
+        {{- 'Current thinking mode: adaptive. You are encouraged to think for complex decision-making, multi-step reasoning, or when analyzing function/tool results.\n' }}
+    {%- endif -%}
+    {{- '</thinking_instructions>' }}
+{%- endmacro -%}
+{%- macro build_developer_message(developer_message) -%}
+    {%- if developer_message and developer_message.content -%}
+        {{- visible_text(developer_message.content) }}
+    {%- else -%}
+        {%- if model_identity is not defined -%}
+            {%- set model_identity = "You are a helpful assistant." -%}
+        {%- endif -%}
+        {{- model_identity }}
+    {%- endif -%}
+{%- endmacro -%}
+{#- Main Template Logic ================================================= -#}
+{#- Role mapping: root -> system sp (high priority), system/developer -> developer sp (low priority) -#}
+{%- set system_message = none -%}
+{%- set developer_message = none -%}
+{%- set conversation_messages = messages -%}
+{%- if messages and messages[0].role == "root" -%}
+    {%- set system_message = messages[0] -%}
+    {%- set conversation_messages = messages[1:] -%}
+    {%- if conversation_messages and conversation_messages[0].role in ["system", "developer"] -%}
+        {%- set developer_message = conversation_messages[0] -%}
+        {%- set conversation_messages = conversation_messages[1:] -%}
+    {%- endif -%}
+{%- elif messages and messages[0].role in ["system", "developer"] -%}
+    {%- set developer_message = messages[0] -%}
+    {%- set conversation_messages = messages[1:] -%}
+{%- endif -%}
+{#- Render system sp (higher priority, root role only) -#}
+{{- bod_token ~ bos_token ~ 'system' ~ '\n' }}
+{{- build_system_message(system_message) }}
+{{- eos_token ~ '\n' }}
+
+{#- Render developer sp (lower priority: system/developer role + tools) -#}
+{{- bos_token ~ 'developer' ~ '\n' }}
+{{- build_developer_message(developer_message) }}
+{%- if tools -%}
+    {{- '\n\n' ~ '# Tools' ~ '\n' ~ 'You may call one or more tools to assist with the user query.\nHere are the tools available in JSONSchema format:' ~ '\n' }}
+    {{- '\n' ~ '<tools>' ~ '\n' }}
+    {{- render_tool_namespace("functions", tools) }}
+    {{- '</tools>' ~ '\n\n' }}
+    {{- 'To call tools, wrap all invocations in a single ' ~ toolcall_begin_token ~ toolcall_end_token ~ ' block. Parameter values containing nested objects or arrays are recursively expanded into XML elements. Example:\n' }}
+    {{- '\n' ~ toolcall_begin_token ~ '\n' }}
+    {{- ns_token + '<invoke name="tool-name-1">' }}
+    {{- ns_token + '<param-1>value-1' + ns_token + '</param-1>' }}
+    {{- ns_token + '<param-2>' }}
+    {{- ns_token + '<item>' }}
+    {{- ns_token + '<key-a>val-a' + ns_token + '</key-a>' }}
+    {{- ns_token + '<key-b>val-b' + ns_token + '</key-b>' }}
+    {{- ns_token + '</item>' }}
+    {{- ns_token + '</param-2>' }}
+    {{- ns_token + '</invoke>\n' }}
+    {{- ns_token + '<invoke name="tool-name-2">' }}
+    {{- ns_token + '<param-1>value-1' + ns_token + '</param-1>' }}
+    {{- ns_token + '</invoke>\n' }}
+    {{- toolcall_end_token }}
+{%- endif -%}
+{{- eos_token ~ '\n' }}
+
+{#- Render messages -#}
+{%- set last_tool_call = namespace(name=none) -%}
+{%- for message in conversation_messages -%}
+    {%- if message.role == 'assistant' -%}
+        {{- bos_token ~ 'ai' ~ '\n' }}
+
+        {%- set reasoning_content = '' %}
+        {%- set content = visible_text(message.content) %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if think_end_token in content %}
+                {%- set reasoning_content = content.split(think_end_token)[0].strip('\n').split(think_begin_token)[-1].strip('\n') %}
+                {%- set content = content.split(think_end_token)[-1].strip('\n') %}
+            {%- endif %}
+        {%- endif %}
+
+        {%- if reasoning_content -%}
+            {#- Render thinking for every assistant turn (all-turn visible) -#}
+            {{- think_begin_token ~ reasoning_content ~ think_end_token }}
+        {%- else -%}
+            {#- No thinking rendered → prefix with think_end_token -#}
+            {{- think_end_token }}
+        {%- endif -%}
+
+        {%- if content -%}
+            {{- content }}
+        {%- endif -%}
+        {%- if message.tool_calls -%}
+            {{- toolcall_begin_token ~ '\n' }}
+
+            {%- for tool_call in message.tool_calls -%}
+                {%- if tool_call.function -%}
+                    {%- set tool_call = tool_call.function -%}
+                {%- endif -%}
+{{- ns_token + '<invoke name="' + tool_call.name + '">' }}
+{%- set _args = tool_call.arguments -%}
+{%- for k, v in _args.items() if v is not none %}
+{{- ns_token + '<' + k + '>' -}}
+{{- to_xml(v, ns_token) -}}
+{{- ns_token + '</' + k + '>' }}
+{%- endfor -%}
+{{- ns_token + '</invoke>' ~ '\n' }}
+            {%- endfor -%}
+
+            {{- toolcall_end_token }}
+            {%- if message.tool_calls[-1].function -%}
+                {%- set last_tool_call.name = message.tool_calls[-1].function.name -%}
+            {%- else -%}
+                {%- set last_tool_call.name = message.tool_calls[-1].name -%}
+            {%- endif -%}
+        {%- else -%}
+            {%- set last_tool_call.name = none -%}
+        {%- endif -%}
+        {{- eos_token ~ '\n' }}
+
+    {%- elif message.role == 'tool' -%}
+        {%- if last_tool_call.name is none -%}
+            {{- raise_exception("Message has tool role, but there was no previous assistant message with a tool call!") }}
+        {%- endif -%}
+        {%- if loop.first or (conversation_messages[loop.index0 - 1].role != 'tool') -%}
+            {{- bos_token ~ 'tool' }}
+        {%- endif -%}
+        {{- '\n<response>' }}
+        {%- if message.content is string -%}
+            {{- message.content }}
+        {%- else -%}
+            {%- for tr in message.content -%}
+                {%- if tr is mapping and tr.type is defined and tr.type == 'image' -%}
+                    {{- image_token }}
+                {%- elif tr is mapping and tr.type is defined and tr.type == 'video' -%}
+                    {{- video_token }}
+                {%- else -%}
+                    {{- tr.output if tr.output is defined else (tr.text if tr.type == 'text' and tr.text is defined else tr) }}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
+        {{- '</response>' }}
+        {%- if loop.last or (conversation_messages[loop.index0 + 1].role != 'tool') -%}
+            {{- eos_token ~ '\n' -}}
+        {%- endif -%}
+
+    {%- elif message.role == 'user' -%}
+        {{- bos_token ~ 'user' ~ '\n' }}
+        {{- visible_text(message.content) }}
+        {{- eos_token ~ '\n' }}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- Generation prompt -#}
+{%- if add_generation_prompt -%}
+{{- bos_token ~ 'ai' ~ '\n' }}
+{%- if thinking_mode is defined and thinking_mode == "disabled" -%}
+    {{- think_end_token }}
+{%- elif thinking_mode is defined and thinking_mode == "adaptive" -%}
+    {#- adaptive: no prefix, let model decide -#}
+{%- elif thinking_mode is defined and thinking_mode == "enabled" -%}
+    {#- enabled or not defined: default to think -#}
+    {{- think_begin_token }}
+{%- else -%}
+    {#- adaptive: no prefix, let model decide -#}
+{%- endif -%}
+{%- endif -%}

config.json

@@ -0,0 +1,431 @@
+{
+ "architectures": [
+  "MiniMaxM3SparseForConditionalGeneration"
+ ],
+ "auto_map": {
+  "AutoConfig": "configuration_minimax_m3_vl.MiniMaxM3VLConfig"
+ },
+ "model_type": "minimax_m3_vl",
+ "text_config": {
+  "hidden_size": 6144,
+  "intermediate_size": 3072,
+  "num_hidden_layers": 60,
+  "num_attention_heads": 64,
+  "num_key_value_heads": 4,
+  "head_dim": 128,
+  "vocab_size": 200064,
+  "max_position_embeddings": 1048576,
+  "rms_norm_eps": 1e-06,
+  "use_gemma_norm": true,
+  "attention_output_gate": false,
+  "rope_theta": 5000000,
+  "rotary_dim": 64,
+  "partial_rotary_factor": 0.5,
+  "hidden_act": "swigluoai",
+  "use_qk_norm": true,
+  "tie_word_embeddings": false,
+  "dense_intermediate_size": 12288,
+  "shared_intermediate_size": 3072,
+  "num_local_experts": 128,
+  "num_experts_per_tok": 4,
+  "n_shared_experts": 1,
+  "scoring_func": "sigmoid",
+  "use_routing_bias": true,
+  "moe_layer_freq": [
+   0,
+   0,
+   0,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1,
+   1
+  ],
+  "qk_norm_type": "per_head",
+  "num_mtp_modules": 7,
+  "num_nextn_predict_layers": 1,
+  "swiglu_alpha": 1.702,
+  "swiglu_limit": 7.0,
+  "routed_scaling_factor": 2.0,
+  "sparse_attention_config": {
+   "use_sparse_attention": true,
+   "sparse_index_dim": 128,
+   "sparse_num_index_heads": 4,
+   "sparse_topk_blocks": 16,
+   "sparse_block_size": 128,
+   "sparse_disable_index_value": [
+    0,
+    0,
+    0,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1
+   ],
+   "sparse_score_type": "max",
+   "sparse_init_block": 0,
+   "sparse_local_block": 1,
+   "sparse_attention_freq": [
+    0,
+    0,
+    0,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1,
+    1
+   ]
+  },
+  "architectures": [
+   "MiniMaxM3SparseForCausalLM"
+  ]
+ },
+ "vision_config": {
+  "hidden_size": 1280,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 32,
+  "intermediate_size": 5120,
+  "patch_size": 14,
+  "image_size": 2016,
+  "projection_dim": 6144,
+  "position_embedding_type": "rope",
+  "rope_mode": "3d",
+  "rope_theta": 10000.0,
+  "attention_dropout": 0.0,
+  "hidden_act": "gelu",
+  "initializer_factor": 1.0,
+  "initializer_range": 0.02,
+  "layer_norm_eps": 1e-05,
+  "model_type": "clip_vision_model",
+  "num_channels": 3,
+  "vocab_size": 32000,
+  "img_token_compression_config": {
+   "image_token_compression_method": "patch_merge",
+   "spatial_merge_size": 2,
+   "temporal_patch_size": 2
+  },
+  "vision_segment_max_frames": 4
+ },
+ "img_token_compression_config": {
+  "image_token_compression_method": "patch_merge",
+  "spatial_merge_size": 2,
+  "temporal_patch_size": 2
+ },
+ "image_grid_pinpoints": "[(336, 336), (336, 672), (336, 1008), (336, 1344), (336, 1680), (336, 2016), (672, 336), (672, 672), (672, 1008), (672, 1344), (672, 1680), (672, 2016), (1008, 336), (1008, 672), (1008, 1008), (1008, 1344), (1008, 1680), (1008, 2016), (1344, 336), (1344, 672), (1344, 1008), (1344, 1344), (1344, 1680), (1344, 2016), (1680, 336), (1680, 672), (1680, 1008), (1680, 1344), (1680, 1680), (1680, 2016), (2016, 336), (2016, 672), (2016, 1008), (2016, 1344), (2016, 1680), (2016, 2016)]",
+ "image_seq_length": 576,
+ "image_token_index": 200025,
+ "video_token_index": 200026,
+ "multimodal_projector_bias": true,
+ "num_reward_heads": 0,
+ "process_image_mode": "dynamic_res",
+ "projector_hidden_act": "gelu",
+ "vision_feature_layer": -1,
+ "vision_feature_select_strategy": "full",
+ "torch_dtype": "bfloat16",
+ "transformers_version": "4.52.4",
+ "projector_hidden_size": 6144,
+ "quantization_config": {
+  "config_groups": {
+   "group_0": {
+    "input_activations": {
+     "dynamic": false,
+     "num_bits": 4,
+     "type": "float",
+     "group_size": 16
+    },
+    "weights": {
+     "dynamic": false,
+     "num_bits": 4,
+     "type": "float",
+     "group_size": 16
+    },
+    "targets": [
+     "Linear"
+    ]
+   }
+  },
+  "ignore": [
+   "lm_head",
+   "model.embed_tokens",
+   "vision_tower",
+   "multi_modal_projector",
+   "patch_merge_mlp",
+   "language_model.model.layers.3.block_sparse_moe.gate",
+   "language_model.model.layers.3.self_attn*",
+   "language_model.model.layers.4.block_sparse_moe.gate",
+   "language_model.model.layers.4.self_attn*",
+   "language_model.model.layers.5.block_sparse_moe.gate",
+   "language_model.model.layers.5.self_attn*",
+   "language_model.model.layers.6.block_sparse_moe.gate",
+   "language_model.model.layers.6.self_attn*",
+   "language_model.model.layers.7.block_sparse_moe.gate",
+   "language_model.model.layers.7.self_attn*",
+   "language_model.model.layers.8.block_sparse_moe.gate",
+   "language_model.model.layers.8.self_attn*",
+   "language_model.model.layers.9.block_sparse_moe.gate",
+   "language_model.model.layers.9.self_attn*",
+   "language_model.model.layers.10.block_sparse_moe.gate",
+   "language_model.model.layers.10.self_attn*",
+   "language_model.model.layers.11.block_sparse_moe.gate",
+   "language_model.model.layers.11.self_attn*",
+   "language_model.model.layers.12.block_sparse_moe.gate",
+   "language_model.model.layers.12.self_attn*",
+   "language_model.model.layers.13.block_sparse_moe.gate",
+   "language_model.model.layers.13.self_attn*",
+   "language_model.model.layers.14.block_sparse_moe.gate",
+   "language_model.model.layers.14.self_attn*",
+   "language_model.model.layers.15.block_sparse_moe.gate",
+   "language_model.model.layers.15.self_attn*",
+   "language_model.model.layers.16.block_sparse_moe.gate",
+   "language_model.model.layers.16.self_attn*",
+   "language_model.model.layers.17.block_sparse_moe.gate",
+   "language_model.model.layers.17.self_attn*",
+   "language_model.model.layers.18.block_sparse_moe.gate",
+   "language_model.model.layers.18.self_attn*",
+   "language_model.model.layers.19.block_sparse_moe.gate",
+   "language_model.model.layers.19.self_attn*",
+   "language_model.model.layers.20.block_sparse_moe.gate",
+   "language_model.model.layers.20.self_attn*",
+   "language_model.model.layers.21.block_sparse_moe.gate",
+   "language_model.model.layers.21.self_attn*",
+   "language_model.model.layers.22.block_sparse_moe.gate",
+   "language_model.model.layers.22.self_attn*",
+   "language_model.model.layers.23.block_sparse_moe.gate",
+   "language_model.model.layers.23.self_attn*",
+   "language_model.model.layers.24.block_sparse_moe.gate",
+   "language_model.model.layers.24.self_attn*",
+   "language_model.model.layers.25.block_sparse_moe.gate",
+   "language_model.model.layers.25.self_attn*",
+   "language_model.model.layers.26.block_sparse_moe.gate",
+   "language_model.model.layers.26.self_attn*",
+   "language_model.model.layers.27.block_sparse_moe.gate",
+   "language_model.model.layers.27.self_attn*",
+   "language_model.model.layers.28.block_sparse_moe.gate",
+   "language_model.model.layers.28.self_attn*",
+   "language_model.model.layers.29.block_sparse_moe.gate",
+   "language_model.model.layers.29.self_attn*",
+   "language_model.model.layers.30.block_sparse_moe.gate",
+   "language_model.model.layers.30.self_attn*",
+   "language_model.model.layers.31.block_sparse_moe.gate",
+   "language_model.model.layers.31.self_attn*",
+   "language_model.model.layers.32.block_sparse_moe.gate",
+   "language_model.model.layers.32.self_attn*",
+   "language_model.model.layers.33.block_sparse_moe.gate",
+   "language_model.model.layers.33.self_attn*",
+   "language_model.model.layers.34.block_sparse_moe.gate",
+   "language_model.model.layers.34.self_attn*",
+   "language_model.model.layers.35.block_sparse_moe.gate",
+   "language_model.model.layers.35.self_attn*",
+   "language_model.model.layers.36.block_sparse_moe.gate",
+   "language_model.model.layers.36.self_attn*",
+   "language_model.model.layers.37.block_sparse_moe.gate",
+   "language_model.model.layers.37.self_attn*",
+   "language_model.model.layers.38.block_sparse_moe.gate",
+   "language_model.model.layers.38.self_attn*",
+   "language_model.model.layers.39.block_sparse_moe.gate",
+   "language_model.model.layers.39.self_attn*",
+   "language_model.model.layers.40.block_sparse_moe.gate",
+   "language_model.model.layers.40.self_attn*",
+   "language_model.model.layers.41.block_sparse_moe.gate",
+   "language_model.model.layers.41.self_attn*",
+   "language_model.model.layers.42.block_sparse_moe.gate",
+   "language_model.model.layers.42.self_attn*",
+   "language_model.model.layers.43.block_sparse_moe.gate",
+   "language_model.model.layers.43.self_attn*",
+   "language_model.model.layers.44.block_sparse_moe.gate",
+   "language_model.model.layers.44.self_attn*",
+   "language_model.model.layers.45.block_sparse_moe.gate",
+   "language_model.model.layers.45.self_attn*",
+   "language_model.model.layers.46.block_sparse_moe.gate",
+   "language_model.model.layers.46.self_attn*",
+   "language_model.model.layers.47.block_sparse_moe.gate",
+   "language_model.model.layers.47.self_attn*",
+   "language_model.model.layers.48.block_sparse_moe.gate",
+   "language_model.model.layers.48.self_attn*",
+   "language_model.model.layers.49.block_sparse_moe.gate",
+   "language_model.model.layers.49.self_attn*",
+   "language_model.model.layers.50.block_sparse_moe.gate",
+   "language_model.model.layers.50.self_attn*",
+   "language_model.model.layers.51.block_sparse_moe.gate",
+   "language_model.model.layers.51.self_attn*",
+   "language_model.model.layers.52.block_sparse_moe.gate",
+   "language_model.model.layers.52.self_attn*",
+   "language_model.model.layers.53.block_sparse_moe.gate",
+   "language_model.model.layers.53.self_attn*",
+   "language_model.model.layers.54.block_sparse_moe.gate",
+   "language_model.model.layers.54.self_attn*",
+   "language_model.model.layers.55.block_sparse_moe.gate",
+   "language_model.model.layers.55.self_attn*",
+   "language_model.model.layers.56.block_sparse_moe.gate",
+   "language_model.model.layers.56.self_attn*",
+   "language_model.model.layers.57.block_sparse_moe.gate",
+   "language_model.model.layers.57.self_attn*",
+   "language_model.model.layers.58.block_sparse_moe.gate",
+   "language_model.model.layers.58.self_attn*",
+   "language_model.model.layers.59.block_sparse_moe.gate",
+   "language_model.model.layers.59.self_attn*",
+   "language_model.model.layers.0.*",
+   "language_model.model.layers.1.*",
+   "language_model.model.layers.2.*"
+  ],
+  "quant_algo": "NVFP4",
+  "quant_method": "modelopt"
+ }
+}

configuration_minimax_m3_vl.py

@@ -0,0 +1,111 @@
+"""HuggingFace configs for the MiniMax VL family (M2 VL / M3 VL).
+
+This file is bundled into every converted HF checkpoint so that loading via
+``AutoConfig.from_pretrained(..., trust_remote_code=True)`` works without any
+runtime dependency on sglang or other internal packages — only stock
+``transformers`` is required.
+
+The class definitions intentionally mirror
+``sglang.srt.configs.minimax_vl``; if either side changes, keep them in sync.
+
+The file is named ``configuration_minimax_m3_vl.py`` (matching the legacy
+``model_type="minimax_m3_vl"`` and the converter's ``auto_map`` entry) so
+that ckpts produced by this converter remain loadable by older sglang versions
+that only know the ``MiniMaxM3VL*`` names. The canonical class is
+``MiniMaxM3VLConfig``; ``MiniMaxM3VLConfig`` is a thin BC alias whose only
+purpose is to be referenced from ``auto_map``.
+"""
+
+from typing import Optional
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.models.auto import CONFIG_MAPPING
+
+
+def _coerce_sub_config(
+    sub_config: Optional[dict], default_model_type: str
+) -> Optional[PretrainedConfig]:
+    """Convert a config dict to a ``PretrainedConfig`` instance.
+
+    If ``model_type`` is registered in HF ``CONFIG_MAPPING`` the corresponding
+    config class is used; otherwise we fall back to a generic
+    ``PretrainedConfig`` so all dict keys still become real attributes (M3's
+    text backbone uses ``model_type="minimax_m2"`` which is not in
+    ``CONFIG_MAPPING``).
+    """
+    if not isinstance(sub_config, dict):
+        return sub_config
+    model_type = sub_config.get("model_type", default_model_type)
+    cls = CONFIG_MAPPING.get(model_type, PretrainedConfig)
+    return cls(**sub_config)
+
+
+class MiniMaxVLBaseConfig(PretrainedConfig):
+    """Base config shared by every MiniMax VL variant.
+
+    Handles vision/text sub-config coercion. Concrete subclasses only need to
+    declare a unique ``model_type`` string.
+    """
+
+    def __init__(
+        self,
+        vision_config: Optional[dict] = None,
+        text_config: Optional[dict] = None,
+        image_token_index: int = 200025,
+        video_token_index: int = 200026,
+        image_seq_length: int = 576,
+        process_image_mode: str = "dynamic_res",
+        projector_hidden_act: str = "gelu",
+        multimodal_projector_bias: bool = True,
+        vision_feature_layer: int = -1,
+        vision_feature_select_strategy: str = "full",
+        img_token_compression_config: Optional[dict] = None,
+        image_grid_pinpoints: Optional[str] = None,
+        **kwargs,
+    ):
+        self.vision_config = _coerce_sub_config(vision_config, "clip_vision_model")
+        self.text_config = _coerce_sub_config(text_config, "mixtral")
+
+        self.image_token_index = image_token_index
+        self.video_token_index = video_token_index
+        self.image_seq_length = image_seq_length
+        self.process_image_mode = process_image_mode
+        self.projector_hidden_act = projector_hidden_act
+        self.multimodal_projector_bias = multimodal_projector_bias
+        self.vision_feature_layer = vision_feature_layer
+        self.vision_feature_select_strategy = vision_feature_select_strategy
+        self.img_token_compression_config = img_token_compression_config or {}
+        self.image_grid_pinpoints = image_grid_pinpoints
+
+        super().__init__(**kwargs)
+
+    def __post_init__(self, **kwargs):
+        super().__post_init__(**kwargs)
+        if hasattr(self, "vision_config"):
+            self.vision_config = _coerce_sub_config(self.vision_config, "clip_vision_model")
+        if hasattr(self, "text_config"):
+            self.text_config = _coerce_sub_config(self.text_config, "mixtral")
+
+
+class MiniMaxM2VLConfig(MiniMaxVLBaseConfig):
+    """MiniMax M2 VL: vision tower + M2 (Mixtral-style MoE) text backbone."""
+
+    model_type = "minimax_m2_vl"
+
+
+class MiniMaxM3VLConfig(MiniMaxVLBaseConfig):
+    """MiniMax M3 VL: vision tower + M3 (mixed sparse/dense MoE) text backbone."""
+
+    model_type = "minimax_m3_vl"
+
+
+class MiniMaxM2MiniVLConfig(MiniMaxM2VLConfig):
+    """Legacy alias kept so old ``model_type="minimax_m2_mini_vl"`` ckpts load."""
+
+    model_type = "minimax_m2_mini_vl"
+
+
+class MiniMaxM3VLConfig(MiniMaxM3VLConfig):
+    """Legacy alias kept so old ``model_type="minimax_m3_vl"`` ckpts load."""
+
+    model_type = "minimax_m3_vl"

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "bos_token_id": 200019,
+  "do_sample": true,
+  "eos_token_id": 200020,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "4.46.1"
+}

hf_quant_config.json

@@ -0,0 +1,135 @@
+{
+ "producer": {
+  "name": "modelopt",
+  "version": "0.44.0"
+ },
+ "quantization": {
+  "quant_algo": "NVFP4",
+  "kv_cache_quant_algo": null,
+  "group_size": 16,
+  "exclude_modules": [
+   "lm_head",
+   "model.embed_tokens",
+   "vision_tower",
+   "multi_modal_projector",
+   "patch_merge_mlp",
+   "language_model.model.layers.3.block_sparse_moe.gate",
+   "language_model.model.layers.3.self_attn*",
+   "language_model.model.layers.4.block_sparse_moe.gate",
+   "language_model.model.layers.4.self_attn*",
+   "language_model.model.layers.5.block_sparse_moe.gate",
+   "language_model.model.layers.5.self_attn*",
+   "language_model.model.layers.6.block_sparse_moe.gate",
+   "language_model.model.layers.6.self_attn*",
+   "language_model.model.layers.7.block_sparse_moe.gate",
+   "language_model.model.layers.7.self_attn*",
+   "language_model.model.layers.8.block_sparse_moe.gate",
+   "language_model.model.layers.8.self_attn*",
+   "language_model.model.layers.9.block_sparse_moe.gate",
+   "language_model.model.layers.9.self_attn*",
+   "language_model.model.layers.10.block_sparse_moe.gate",
+   "language_model.model.layers.10.self_attn*",
+   "language_model.model.layers.11.block_sparse_moe.gate",
+   "language_model.model.layers.11.self_attn*",
+   "language_model.model.layers.12.block_sparse_moe.gate",
+   "language_model.model.layers.12.self_attn*",
+   "language_model.model.layers.13.block_sparse_moe.gate",
+   "language_model.model.layers.13.self_attn*",
+   "language_model.model.layers.14.block_sparse_moe.gate",
+   "language_model.model.layers.14.self_attn*",
+   "language_model.model.layers.15.block_sparse_moe.gate",
+   "language_model.model.layers.15.self_attn*",
+   "language_model.model.layers.16.block_sparse_moe.gate",
+   "language_model.model.layers.16.self_attn*",
+   "language_model.model.layers.17.block_sparse_moe.gate",
+   "language_model.model.layers.17.self_attn*",
+   "language_model.model.layers.18.block_sparse_moe.gate",
+   "language_model.model.layers.18.self_attn*",
+   "language_model.model.layers.19.block_sparse_moe.gate",
+   "language_model.model.layers.19.self_attn*",
+   "language_model.model.layers.20.block_sparse_moe.gate",
+   "language_model.model.layers.20.self_attn*",
+   "language_model.model.layers.21.block_sparse_moe.gate",
+   "language_model.model.layers.21.self_attn*",
+   "language_model.model.layers.22.block_sparse_moe.gate",
+   "language_model.model.layers.22.self_attn*",
+   "language_model.model.layers.23.block_sparse_moe.gate",
+   "language_model.model.layers.23.self_attn*",
+   "language_model.model.layers.24.block_sparse_moe.gate",
+   "language_model.model.layers.24.self_attn*",
+   "language_model.model.layers.25.block_sparse_moe.gate",
+   "language_model.model.layers.25.self_attn*",
+   "language_model.model.layers.26.block_sparse_moe.gate",
+   "language_model.model.layers.26.self_attn*",
+   "language_model.model.layers.27.block_sparse_moe.gate",
+   "language_model.model.layers.27.self_attn*",
+   "language_model.model.layers.28.block_sparse_moe.gate",
+   "language_model.model.layers.28.self_attn*",
+   "language_model.model.layers.29.block_sparse_moe.gate",
+   "language_model.model.layers.29.self_attn*",
+   "language_model.model.layers.30.block_sparse_moe.gate",
+   "language_model.model.layers.30.self_attn*",
+   "language_model.model.layers.31.block_sparse_moe.gate",
+   "language_model.model.layers.31.self_attn*",
+   "language_model.model.layers.32.block_sparse_moe.gate",
+   "language_model.model.layers.32.self_attn*",
+   "language_model.model.layers.33.block_sparse_moe.gate",
+   "language_model.model.layers.33.self_attn*",
+   "language_model.model.layers.34.block_sparse_moe.gate",
+   "language_model.model.layers.34.self_attn*",
+   "language_model.model.layers.35.block_sparse_moe.gate",
+   "language_model.model.layers.35.self_attn*",
+   "language_model.model.layers.36.block_sparse_moe.gate",
+   "language_model.model.layers.36.self_attn*",
+   "language_model.model.layers.37.block_sparse_moe.gate",
+   "language_model.model.layers.37.self_attn*",
+   "language_model.model.layers.38.block_sparse_moe.gate",
+   "language_model.model.layers.38.self_attn*",
+   "language_model.model.layers.39.block_sparse_moe.gate",
+   "language_model.model.layers.39.self_attn*",
+   "language_model.model.layers.40.block_sparse_moe.gate",
+   "language_model.model.layers.40.self_attn*",
+   "language_model.model.layers.41.block_sparse_moe.gate",
+   "language_model.model.layers.41.self_attn*",
+   "language_model.model.layers.42.block_sparse_moe.gate",
+   "language_model.model.layers.42.self_attn*",
+   "language_model.model.layers.43.block_sparse_moe.gate",
+   "language_model.model.layers.43.self_attn*",
+   "language_model.model.layers.44.block_sparse_moe.gate",
+   "language_model.model.layers.44.self_attn*",
+   "language_model.model.layers.45.block_sparse_moe.gate",
+   "language_model.model.layers.45.self_attn*",
+   "language_model.model.layers.46.block_sparse_moe.gate",
+   "language_model.model.layers.46.self_attn*",
+   "language_model.model.layers.47.block_sparse_moe.gate",
+   "language_model.model.layers.47.self_attn*",
+   "language_model.model.layers.48.block_sparse_moe.gate",
+   "language_model.model.layers.48.self_attn*",
+   "language_model.model.layers.49.block_sparse_moe.gate",
+   "language_model.model.layers.49.self_attn*",
+   "language_model.model.layers.50.block_sparse_moe.gate",
+   "language_model.model.layers.50.self_attn*",
+   "language_model.model.layers.51.block_sparse_moe.gate",
+   "language_model.model.layers.51.self_attn*",
+   "language_model.model.layers.52.block_sparse_moe.gate",
+   "language_model.model.layers.52.self_attn*",
+   "language_model.model.layers.53.block_sparse_moe.gate",
+   "language_model.model.layers.53.self_attn*",
+   "language_model.model.layers.54.block_sparse_moe.gate",
+   "language_model.model.layers.54.self_attn*",
+   "language_model.model.layers.55.block_sparse_moe.gate",
+   "language_model.model.layers.55.self_attn*",
+   "language_model.model.layers.56.block_sparse_moe.gate",
+   "language_model.model.layers.56.self_attn*",
+   "language_model.model.layers.57.block_sparse_moe.gate",
+   "language_model.model.layers.57.self_attn*",
+   "language_model.model.layers.58.block_sparse_moe.gate",
+   "language_model.model.layers.58.self_attn*",
+   "language_model.model.layers.59.block_sparse_moe.gate",
+   "language_model.model.layers.59.self_attn*",
+   "language_model.model.layers.0.*",
+   "language_model.model.layers.1.*",
+   "language_model.model.layers.2.*"
+  ]
+ }
+}

image_processor.py

@@ -0,0 +1,223 @@
+# Copyright 2023-2024 SGLang Team
+# Licensed under the Apache License, Version 2.0 (the "License");
+"""
+MiniMax VL family HuggingFace-compatible Processor, ImageProcessor, VideoProcessor.
+"""
+import math
+from typing import List, Tuple
+
+import torch
+from torchvision.transforms import InterpolationMode
+from transformers import BatchFeature
+from transformers.image_processing_utils_fast import (
+    BaseImageProcessorFast,
+    group_images_by_shape,
+    reorder_images,
+)
+from transformers.image_utils import PILImageResampling, SizeDict
+from transformers.processing_utils import (
+    ImagesKwargs,
+    Unpack,
+)
+from transformers.utils import TensorType
+
+MAX_RATIO = 200
+
+
+def round_by_factor(number: int, factor: int) -> int:
+    return round(number / factor) * factor
+
+
+def ceil_by_factor(number: int, factor: int) -> int:
+    return math.ceil(number / factor) * factor
+
+
+def floor_by_factor(number: int, factor: int) -> int:
+    return math.floor(number / factor) * factor
+
+
+def smart_resize(
+    height: int,
+    width: int,
+    factor: int = 28,
+    min_pixels: int = 4 * 28 * 28,
+    max_pixels: int = 451584,
+) -> tuple[int, int]:
+    if max(height, width) / min(height, width) > MAX_RATIO:
+        raise ValueError(
+            f"absolute aspect ratio must be smaller than {MAX_RATIO}, "
+            f"got {max(height, width) / min(height, width)}"
+        )
+    h_bar = max(factor, round_by_factor(height, factor))
+    w_bar = max(factor, round_by_factor(width, factor))
+    if h_bar * w_bar > max_pixels:
+        beta = math.sqrt((height * width) / max_pixels)
+        h_bar = floor_by_factor(height / beta, factor)
+        w_bar = floor_by_factor(width / beta, factor)
+    elif h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (height * width))
+        h_bar = ceil_by_factor(height * beta, factor)
+        w_bar = ceil_by_factor(width * beta, factor)
+    return h_bar, w_bar
+
+
+# ==============================================================================
+# MiniMax M3 VL Image Processor Fast (Fast Mode - Torch based)
+# ==============================================================================
+
+
+class MiniMaxM3VLImageProcessorKwargs(ImagesKwargs, total=False):
+    patch_size: int
+    temporal_patch_size: int
+    merge_size: int
+    max_pixels: int
+
+
+class MiniMaxM3VLImageProcessor(BaseImageProcessorFast):
+    do_resize = True
+    resample = PILImageResampling.BICUBIC
+    size = {"height": 672, "width": 672}  # required by base class validation, not used as resize bound
+    default_to_square = False
+    do_rescale = True
+    rescale_factor = 1 / 255
+    do_normalize = True
+    image_mean = [0.48145466, 0.4578275, 0.40821073]
+    image_std = [0.26862954, 0.26130258, 0.27577711]
+    do_convert_rgb = True
+    patch_size = 14
+    temporal_patch_size = 2
+    merge_size = 2
+    max_pixels = 451584             # 672*672
+    valid_kwargs = MiniMaxM3VLImageProcessorKwargs
+    model_input_names = ["pixel_values", "image_grid_thw"]
+
+    def __init__(self, **kwargs: Unpack[MiniMaxM3VLImageProcessorKwargs]):
+        super().__init__(**kwargs)
+
+    def preprocess(
+        self, images, **kwargs: Unpack[MiniMaxM3VLImageProcessorKwargs]
+    ) -> BatchFeature:
+        return super().preprocess(images, **kwargs)
+
+    def _preprocess(
+        self,
+        images: List[torch.Tensor],
+        do_resize: bool,
+        size: SizeDict,
+        resample: PILImageResampling | InterpolationMode | int | None,
+        do_rescale: bool,
+        rescale_factor: float,
+        do_normalize: bool,
+        image_mean: float | List[float] | None,
+        image_std: float | List[float] | None,
+        patch_size: int,
+        temporal_patch_size: int,
+        merge_size: int,
+        max_pixels: int,
+        disable_grouping: bool | None,
+        return_tensors: str | TensorType | None,
+        **kwargs,
+    ) -> BatchFeature:
+        grouped_images, grouped_images_index = group_images_by_shape(
+            images, disable_grouping=disable_grouping
+        )
+        resized_images_grouped = {}
+        factor = patch_size * merge_size
+        for shape, stacked_images in grouped_images.items():
+            height, width = stacked_images.shape[-2:]
+            if do_resize:
+                resized_height, resized_width = smart_resize(
+                    height, width, factor=factor,
+                    max_pixels=max_pixels,
+                )
+                stacked_images = self.resize(
+                    stacked_images,
+                    size=SizeDict(height=resized_height, width=resized_width),
+                    resample=resample,
+                )
+            resized_images_grouped[shape] = stacked_images
+
+        resized_images = reorder_images(resized_images_grouped, grouped_images_index)
+
+        grouped_images, grouped_images_index = group_images_by_shape(
+            resized_images, disable_grouping=disable_grouping
+        )
+        processed_images_grouped = {}
+        processed_grids = {}
+
+        for shape, stacked_images in grouped_images.items():
+            resized_height, resized_width = stacked_images.shape[-2:]
+
+            patches = self.rescale_and_normalize(
+                stacked_images,
+                do_rescale,
+                rescale_factor,
+                do_normalize,
+                image_mean,
+                image_std,
+            )
+            if patches.ndim == 4:
+                patches = patches.unsqueeze(1)
+
+            if patches.shape[1] % temporal_patch_size != 0:
+                repeats = patches[:, -1:].repeat(
+                    1,
+                    temporal_patch_size - (patches.shape[1] % temporal_patch_size),
+                    1,
+                    1,
+                    1,
+                )
+                patches = torch.cat([patches, repeats], dim=1)
+
+            batch_size, grid_t, channel = patches.shape[:3]
+            grid_t = grid_t // temporal_patch_size
+            grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+
+            patches = patches.view(
+                batch_size,
+                grid_t,
+                temporal_patch_size,
+                channel,
+                grid_h // merge_size,
+                merge_size,
+                patch_size,
+                grid_w // merge_size,
+                merge_size,
+                patch_size,
+            )
+            patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
+
+            flatten_patches = patches.reshape(
+                batch_size,
+                grid_t * grid_h * grid_w,
+                channel * temporal_patch_size * patch_size * patch_size,
+            )
+
+            processed_images_grouped[shape] = flatten_patches
+            processed_grids[shape] = [[grid_t, grid_h, grid_w]] * batch_size
+
+        processed_images = reorder_images(
+            processed_images_grouped, grouped_images_index
+        )
+        processed_grids = reorder_images(processed_grids, grouped_images_index)
+
+        pixel_values = torch.cat(processed_images, dim=0)
+        image_grid_thw = torch.tensor(processed_grids, dtype=torch.long)
+
+        return BatchFeature(
+            data={"pixel_values": pixel_values, "image_grid_thw": image_grid_thw},
+            tensor_type=return_tensors,
+        )
+
+    def get_number_of_image_patches(self, height: int, width: int, images_kwargs=None):
+        images_kwargs = images_kwargs or {}
+        patch_size = images_kwargs.get("patch_size", self.patch_size)
+        merge_size = images_kwargs.get("merge_size", self.merge_size)
+        max_pixels = images_kwargs.get("max_pixels", self.max_pixels)
+
+        resized_height, resized_width = smart_resize(
+            height, width, factor=patch_size * merge_size,
+            max_pixels=max_pixels,
+        )
+        grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+        return grid_h * grid_w

model-00001-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3e7d72ba0de5bb55f48da9f438663bf9ac7e01ba23f2c24ccbe95f4737e815f
+size 16106357520

model-00002-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a3ccc511702a9f06382aa77865db578a6ff1d3331aea9126d8289ec2001adc3
+size 16040620504

model-00003-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48f1d297e3e18b9168d8eebfb26fb1ecfb8dcc572eaecad336a05234f571bcdc
+size 16102725336

model-00004-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:519eb2760973b7ab63972551a6db73bd9d868fcae89aebecadffba1ce9dd305a
+size 16101953448

model-00005-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32c74c5f75f5f6f4f2358a117ca56ea0d2a9fb230edf8d357f71d18babd8251d
+size 16034333072

model-00006-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32f27f29e3252105ccc20a273070b2072ba16086e0afa7a2727a6467e686be70
+size 16106651408

model-00007-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7424c3cf546bdbefd03c6eea75005a79359ee1b15f86b179e1da2c6a25e9d9c3
+size 16105109184

model-00008-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41a30539511453248be8bff050bef77de62828b5cec531f7cf5b5dc9846c99f5
+size 16103927448

model-00009-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:441f2237f5319710aac480149832bee84d5f3107a8b42a51ef5c8033ceffa95c
+size 16101942360

model-00010-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a102ed6255f7aa3fa20390163f2ea2ca45808b5b2303da2c41fcafcc3c1f817
+size 16103930568

model-00011-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:741a1739b84e71cc8a05b1d7fa58d899ed616a4e21eb242af56f99c3213ea859
+size 16103928640

model-00012-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa4ba3c63614cda1a8012a1d0f7cf8e9e9ae79c2f7b49cb9fbf2c61b769b7730
+size 16055566328

model-00013-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:118c6f9ea3c67f35e9b6e737a371c6ab66a93173438f3c1a0990ffbcea73d25c
+size 16106652488

model-00014-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebc57aac69600e127519138785b8066ce3c8cb752beee620233bb02c639e65e6
+size 16105109656

model-00015-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77af9a60bc4d2ddfd3061a5d2da8e89ec6ada01863cb1ac6e02cf8d25c3fca6f
+size 16103925720

model-00016-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94e2c3022cb33a6f38db119b79014f3986e63f83b81f71d307d3ca9e35831207
+size 14300568744

preprocessor_config.json

@@ -0,0 +1,32 @@
+{
+  "processor_class": "MiniMaxVLProcessor",
+  "auto_map": {
+    "AutoImageProcessor": "image_processor.MiniMaxM3VLImageProcessor",
+    "AutoProcessor": "processing_minimax.MiniMaxVLProcessor",
+    "AutoVideoProcessor": "video_processor.MiniMaxM3VLVideoProcessor"
+  },
+  "process_image_mode": "dynamic_res",
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "size": [
+    672,
+    672
+  ],
+  "patch_size": 14,
+  "img_token_compression_config": {
+    "image_token_compression_threshold": 1.1,
+    "image_token_compression_method": "patch_merge",
+    "max_image_resolution": 1008,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "add_start_end_special_tokens": true
+}

processing_minimax.py

@@ -0,0 +1,254 @@
+# Copyright 2023-2024 SGLang Team
+# Licensed under the Apache License, Version 2.0 (the "License");
+"""
+MiniMax VL family HuggingFace-compatible Processor, ImageProcessor, VideoProcessor.
+"""
+
+import math
+import re
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torchvision
+from torchvision.transforms import InterpolationMode
+from transformers import BatchFeature
+from transformers.image_processing_utils_fast import (
+    BaseImageProcessorFast,
+    group_images_by_shape,
+    reorder_images,
+)
+from transformers.image_utils import PILImageResampling, SizeDict
+from transformers.processing_utils import (
+    ImagesKwargs,
+    ProcessingKwargs,
+    ProcessorMixin,
+    Unpack,
+    VideosKwargs,
+)
+from transformers.utils import TensorType
+from transformers.video_processing_utils import BaseVideoProcessor
+from transformers.video_utils import group_videos_by_shape, reorder_videos
+
+
+class MiniMaxVLProcessorKwargs(ProcessingKwargs, total=False):
+    _defaults = {
+        "videos_kwargs": {
+            "do_resize": False,
+            "return_metadata": True,
+        },
+    }
+
+
+class MiniMaxVLProcessor(ProcessorMixin):
+    IMAGE_TOKEN = "]<]image[>["
+    VIDEO_TOKEN = "]<]video[>["
+    VISION_START_TOKEN = "]<]start of image[>["
+    VISION_END_TOKEN = "]<]end of image[>["
+
+    def __init__(
+        self, image_processor=None, tokenizer=None, video_processor=None, **kwargs
+    ):
+        self.image_token_id = tokenizer.convert_tokens_to_ids(self.IMAGE_TOKEN)
+        self.video_token_id = tokenizer.convert_tokens_to_ids(self.VIDEO_TOKEN)
+        super().__init__(image_processor, tokenizer, video_processor)
+        # Video expansion also uses image start/end tokens. Separate video
+        # start/end tokens exist in the tokenizer, but the original MiniMax
+        # serving path did not use them; keep that behavior for compatibility.
+        self.vision_start_token_id = tokenizer.convert_tokens_to_ids(
+            self.VISION_START_TOKEN
+        )
+        self.vision_end_token_id = tokenizer.convert_tokens_to_ids(
+            self.VISION_END_TOKEN
+        )
+
+    def _prune_video_tokens(
+        self,
+        input_text: str,
+        video_segments: List[int],
+        video_token: str,
+    ) -> str:
+        """
+        Prune video tokens by temporal_patch_size (e.g., 2:1).
+
+        Expects the prompt to carry exactly sum(video_segments) video
+        tokens — i.e. one token per *sampled* frame. Then drops token.
+
+        Args:
+            input_text: prompt with N video_tokens per segment
+            video_segments: actual sampled frame count per video segment
+            video_token: the video token string, e.g. ']<]video[>['
+
+        Returns:
+            Pruned input_text with ~N/temporal_patch_size tokens per segment.
+        """
+        # If no videos or temporal_patch_size <= 1, no pruning needed
+        if not video_segments or self.video_processor.temporal_patch_size <= 1:
+            return input_text
+
+        # Split while keeping delimiters
+        special_tokens = [video_token]  # , image_token]
+        pattern = "|".join(map(re.escape, special_tokens))
+        parts = re.split(f"({pattern})", input_text)
+
+        def is_timestamp(text: str) -> bool:
+            """Check if text ends with timestamp format like ']<]0.0 seconds[>['"""
+            return (
+                text.endswith("seconds[>[")
+                or text.endswith("seconds[>[ ")
+                or text.endswith("seconds [>[")
+                or text.endswith("seconds [>[ ")
+            )
+
+        def extract_timestamp(text: str) -> str:
+            """Extract timestamp text from the end, starting from ']<]'"""
+            start_index = text.rfind("]<]")
+            if start_index == -1:
+                raise ValueError(f"Failed to extract timestamp: {text}")
+            return text[start_index:]
+
+        # Build new text with pruned video tokens
+        final_parts = []
+        current_seg_idx = 0  # Which video segment we're in
+        frame_in_seg = 0  # Frame index within current segment
+        last_timestamp_len = 0  # Length of timestamp to potentially remove
+
+        for part in parts:
+            if part == video_token:
+                if current_seg_idx < len(video_segments):
+                    if frame_in_seg % self.video_processor.temporal_patch_size == 0:
+                        # Keep this video token
+                        final_parts.append(part)
+                        frame_in_seg += 1
+                        if frame_in_seg >= video_segments[current_seg_idx]:
+                            current_seg_idx += 1
+                            frame_in_seg = 0
+                        last_timestamp_len = 0
+                    else:
+                        # Skip this video token
+                        frame_in_seg += 1
+                        if frame_in_seg >= video_segments[current_seg_idx]:
+                            current_seg_idx += 1
+                            frame_in_seg = 0
+                        # Remove the timestamp that was already appended
+                        if last_timestamp_len > 0:
+                            # Truncate the last part to remove timestamp
+                            assert len(final_parts) > 0
+                            final_parts[-1] = final_parts[-1][:-last_timestamp_len]
+                            last_timestamp_len = 0
+                else:
+                    # No more video segments, keep as is
+                    final_parts.append(part)
+                    last_timestamp_len = 0
+            else:
+                # Text part
+                final_parts.append(part)
+                # Check if this text ends with a timestamp
+                if is_timestamp(part):
+                    last_timestamp_len = len(extract_timestamp(part))
+                else:
+                    last_timestamp_len = 0
+
+        return "".join(final_parts)
+
+    def __call__(
+        self,
+        images=None,
+        text=None,
+        videos=None,
+        **kwargs: Unpack[MiniMaxVLProcessorKwargs],
+    ) -> BatchFeature:
+        output_kwargs = self._merge_kwargs(
+            MiniMaxVLProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+
+        if images is not None:
+            images_kwargs = output_kwargs["images_kwargs"]
+            image_inputs = self.image_processor(images=images, **images_kwargs)
+            image_grid_thw = image_inputs["image_grid_thw"]
+
+        else:
+            image_inputs = {}
+            image_grid_thw = None
+
+        if videos is not None:
+            videos_kwargs = output_kwargs["videos_kwargs"]
+            video_inputs = self.video_processor(videos=videos, **videos_kwargs)
+            video_grid_thw = video_inputs["video_grid_thw"]
+            if not kwargs.get("return_metadata"):
+                video_metadata = video_inputs.pop("video_metadata")
+            else:
+                video_metadata = video_inputs["video_metadata"]
+        else:
+            video_inputs = {}
+            video_grid_thw = None
+
+        if not isinstance(text, list):
+            text = [text]
+        text = text.copy()
+
+        # Expand image tokens
+        if image_grid_thw is not None:
+            merge_length = self.image_processor.merge_size**2
+            placeholder = "]<]placeholder[>["
+            index = 0
+            for i in range(len(text)):
+                while self.IMAGE_TOKEN in text[i]:
+                    num_tokens = image_grid_thw[index].prod() // merge_length
+                    text[i] = text[i].replace(
+                        self.IMAGE_TOKEN,
+                        self.VISION_START_TOKEN
+                        + placeholder * num_tokens
+                        + self.VISION_END_TOKEN,
+                        1,
+                    )
+                    index += 1
+                text[i] = text[i].replace(placeholder, self.IMAGE_TOKEN)
+
+        # Expand video tokens
+        if video_grid_thw is not None:
+            merge_length = self.image_processor.merge_size**2
+            placeholder = "]<]placeholder[>["
+            index = 0
+            for i in range(len(text)):
+                while self.VIDEO_TOKEN in text[i]:
+                    metadata = video_metadata[index]
+                    grid_t = video_grid_thw[index][0]
+                    frame_seqlen = video_grid_thw[index][1:].prod() // merge_length
+
+                    video_placeholder = ""
+                    for frame_idx in range(grid_t):
+                        if (
+                            metadata.fps is not None
+                            and metadata.frames_indices is not None
+                        ):
+                            ts = (
+                                metadata.frames_indices[
+                                    min(
+                                        frame_idx
+                                        * self.video_processor.temporal_patch_size,
+                                        len(metadata.frames_indices) - 1,
+                                    )
+                                ]
+                                / metadata.fps
+                            )
+                            video_placeholder += f"]<]{ts:.1f} seconds[>["
+                        video_placeholder += (
+                            self.VISION_START_TOKEN
+                            + placeholder * frame_seqlen
+                            + self.VISION_END_TOKEN
+                        )
+
+                    text[i] = text[i].replace(self.VIDEO_TOKEN, video_placeholder, 1)
+                    index += 1
+                text[i] = text[i].replace(placeholder, self.VIDEO_TOKEN)
+
+        # Tokenize
+        return_tensors = output_kwargs["text_kwargs"].pop("return_tensors", None)
+        text_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
+
+        return BatchFeature(
+            data={**text_inputs, **image_inputs, **video_inputs},
+            tensor_type=return_tensors,
+        )

sglang_patch/flashinfer_trtllm.py

@@ -0,0 +1,1239 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, cast
+
+import torch
+from torch.nn import Module
+from torch.nn.parameter import Parameter
+
+# Import to register custom ops for torch.compile compatibility
+from sglang.srt.distributed import get_tp_group
+from sglang.srt.distributed.device_communicators.pynccl_allocator import (
+    is_symmetric_memory_enabled,
+    is_tensor_in_symmetric_mempool,
+    use_symmetric_memory,
+)
+from sglang.srt.environ import envs
+from sglang.srt.layers.dp_attention import is_allocation_symmetric
+from sglang.srt.layers.moe.flashinfer_trtllm_moe import (
+    trtllm_fp8_block_scale_moe_wrapper,
+    trtllm_fp8_block_scale_routed_moe_wrapper,
+    trtllm_fp8_per_tensor_scale_moe_wrapper,
+)
+from sglang.srt.layers.moe.moe_runner.base import (
+    MoeQuantInfo,
+    MoeRunnerConfig,
+    _moe_output_buf,
+    register_fused_func,
+)
+from sglang.srt.layers.quantization.fp8_kernel import (
+    per_token_group_quant_fp8,
+    scaled_fp8_quant,
+)
+from sglang.srt.layers.utils import copy_or_rebind_param
+from sglang.srt.utils.common import (
+    is_cuda_alike,
+    is_flashinfer_available,
+    next_power_of_2,
+)
+
+_SGLANG_EXPERIMENTAL_LORA_OPTI = envs.SGLANG_EXPERIMENTAL_LORA_OPTI.get()
+
+logger = __import__("logging").getLogger(__name__)
+
+
+def round_up_to_multiple(x: int, m: int) -> int:
+    """Round up *x* to the nearest multiple of *m*."""
+    return (x + m - 1) // m * m
+
+
+if TYPE_CHECKING:
+    from sglang.srt.layers.moe.token_dispatcher import (
+        StandardCombineInput,
+        StandardDispatchOutput,
+    )
+
+if is_flashinfer_available():
+    from sglang.srt.layers.quantization.fp4_utils import fp4_quantize
+elif is_cuda_alike():
+    from sglang.jit_kernel.nvfp4 import scaled_fp4_quant as fp4_quantize
+else:
+    fp4_quantize = None
+
+_flashinfer_trtllm_shuffle_row_indices_cache_mxfp8: dict[
+    tuple, dict[str, torch.Tensor]
+] = {}
+
+
+def _is_gated(layer: Module) -> bool:
+    """Return whether the MoE layer uses a gated activation (default True)."""
+    is_gated = (
+        getattr(layer, "moe_runner_config", None) and layer.moe_runner_config.is_gated
+    )
+    return True if is_gated is None else is_gated
+
+
+def _align_fp8_moe_weights(
+    w13: torch.Tensor,
+    w2: torch.Tensor,
+    is_gated: bool,
+    min_alignment: int = 16,
+) -> tuple[torch.Tensor, torch.Tensor, int]:
+    """Pad intermediate size so FlashInfer TRTLLM FP8 kernels' alignment holds.
+
+    Returns (w13, w2, padded_intermediate).
+    """
+    num_experts, hidden_size, intermediate = w2.shape
+
+    padded_intermediate = round_up_to_multiple(intermediate, min_alignment)
+    if padded_intermediate == intermediate:
+        return w13, w2, intermediate
+
+    logger.info(
+        "FP8 MoE: padding intermediate size from %d to %d (alignment=%d)",
+        intermediate,
+        padded_intermediate,
+        min_alignment,
+    )
+
+    up_mult = 2 if is_gated else 1
+    padded_gate_up = up_mult * padded_intermediate
+
+    padded_w13 = w13.new_zeros((num_experts, padded_gate_up, w13.shape[2]))
+    padded_w13[:, : w13.shape[1], :] = w13
+
+    padded_w2 = w2.new_zeros((num_experts, hidden_size, padded_intermediate))
+    padded_w2[:, :, :intermediate] = w2
+
+    return padded_w13, padded_w2, padded_intermediate
+
+
+def align_fp8_moe_weights_for_flashinfer_trtllm(
+    layer: Module, swap_w13_halves: bool = False
+) -> None:
+    """Prepare FP8 MoE weights/scales for FlashInfer TRT-LLM kernels.
+
+    Args:
+        layer: The MoE layer to process.
+        swap_w13_halves: If True, swap W13 halves from [Up, Gate] to [Gate, Up].
+            This is needed for ModelOpt FP8 checkpoints which store weights in
+            [Up, Gate] order, while regular FP8 checkpoints store them in [Gate, Up].
+    """
+    from flashinfer import shuffle_matrix_a
+
+    is_gated = _is_gated(layer)
+
+    w13_weight = cast(torch.Tensor, layer.w13_weight)
+    w2_weight = cast(torch.Tensor, layer.w2_weight)
+    num_experts, gate_up_dim, hidden = w13_weight.shape
+
+    # Optionally swap W13 halves: [Up, Gate] -> [Gate, Up] (only for gated)
+    if swap_w13_halves and is_gated:
+        inter = gate_up_dim // 2
+        w13_weight = (
+            w13_weight.reshape(num_experts, 2, inter, hidden)
+            .flip(dims=[1])
+            .reshape(num_experts, gate_up_dim, hidden)
+        )
+
+    # Pad for kernel alignment (non-gated needs 128, gated needs 16)
+    min_alignment = 16 if is_gated else 128
+    w13_weight, w2_weight, _ = _align_fp8_moe_weights(
+        w13_weight, w2_weight, is_gated, min_alignment
+    )
+    num_experts, gate_up_dim, hidden = w13_weight.shape
+
+    epilogue_tile_m = 128
+
+    if is_gated:
+        from flashinfer import reorder_rows_for_gated_act_gemm
+
+        w13_interleaved_list = [
+            reorder_rows_for_gated_act_gemm(w13_weight[i]) for i in range(num_experts)
+        ]
+        w13_processed: torch.Tensor = torch.stack(w13_interleaved_list).reshape(
+            num_experts, gate_up_dim, hidden
+        )
+    else:
+        w13_processed = w13_weight
+
+    # Shuffle weights for transposed MMA output (both W13, W2)
+    w13_shuffled = [
+        shuffle_matrix_a(w13_processed[i].view(torch.uint8), epilogue_tile_m)
+        for i in range(num_experts)
+    ]
+    w2_shuffled = [
+        shuffle_matrix_a(w2_weight[i].view(torch.uint8), epilogue_tile_m)
+        for i in range(num_experts)
+    ]
+
+    layer.w13_weight = Parameter(
+        torch.stack(w13_shuffled).view(torch.float8_e4m3fn),
+        requires_grad=False,
+    )
+    layer.w2_weight = Parameter(
+        torch.stack(w2_shuffled).view(torch.float8_e4m3fn),
+        requires_grad=False,
+    )
+
+    # Precompute and register per-expert output scaling factors for FI MoE.
+    # Note: w13_input_scale and w2_input_scale are scalar Parameters post-reduction.
+    assert hasattr(layer, "w13_input_scale") and layer.w13_input_scale is not None
+    assert hasattr(layer, "w2_input_scale") and layer.w2_input_scale is not None
+    assert hasattr(layer, "w13_weight_scale") and layer.w13_weight_scale is not None
+    assert hasattr(layer, "w2_weight_scale") and layer.w2_weight_scale is not None
+
+    input_scale = cast(torch.Tensor, layer.w13_input_scale).to(torch.float32)
+    activation_scale = cast(torch.Tensor, layer.w2_input_scale).to(torch.float32)
+    w13_weight_scale = cast(torch.Tensor, layer.w13_weight_scale).to(torch.float32)
+    w2_weight_scale = cast(torch.Tensor, layer.w2_weight_scale).to(torch.float32)
+
+    # For gated (SwiGLU): g1_alphas = w1_scale * a1_scale, g1_scale_c = g1_alphas / a2_scale
+    # For non-gated (Relu2): g1_scale_c = 1 / a2_scale (no gate dequant contribution)
+    if is_gated:
+        output1_scales_scalar = (
+            w13_weight_scale * input_scale * (1.0 / activation_scale)
+        )
+    else:
+        output1_scales_scalar = torch.ones_like(w13_weight_scale) * (
+            1.0 / activation_scale
+        )
+    output1_scales_gate_scalar = w13_weight_scale * input_scale
+    output2_scales_scalar = activation_scale * w2_weight_scale
+
+    layer.output1_scales_scalar = Parameter(output1_scales_scalar, requires_grad=False)
+    layer.output1_scales_gate_scalar = Parameter(
+        output1_scales_gate_scalar, requires_grad=False
+    )
+    layer.output2_scales_scalar = Parameter(output2_scales_scalar, requires_grad=False)
+
+
+def _align_mxfp8_moe_weights(
+    w13: torch.Tensor,
+    w13_scale: torch.Tensor,
+    w2: torch.Tensor,
+    w2_scale: torch.Tensor,
+    is_gated: bool,
+    min_alignment: int = 16,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, int]:
+    """Pad intermediate size so FlashInfer TRTLLM MXFP8 kernels' alignment holds.
+
+    Returns (w13, w13_scale, w2, w2_scale, padded_intermediate).
+    """
+    num_experts, hidden_size, intermediate = w2.shape
+
+    padded_intermediate = round_up_to_multiple(intermediate, min_alignment)
+    if padded_intermediate == intermediate:
+        return w13, w13_scale, w2, w2_scale, intermediate
+
+    logger.info(
+        "MXFP8 MoE: padding intermediate size from %d to %d (alignment=%d)",
+        intermediate,
+        padded_intermediate,
+        min_alignment,
+    )
+
+    up_mult = 2 if is_gated else 1
+    padded_gate_up = up_mult * padded_intermediate
+
+    padded_w13 = w13.new_zeros((num_experts, padded_gate_up, w13.shape[2]))
+    padded_w13[:, : w13.shape[1], :] = w13
+
+    padded_w2 = w2.new_zeros((num_experts, hidden_size, padded_intermediate))
+    padded_w2[:, :, :intermediate] = w2
+
+    padded_w13_scale = w13_scale.new_zeros(
+        (num_experts, padded_gate_up, w13_scale.shape[2])
+    )
+    padded_w13_scale[:, : w13_scale.shape[1], :] = w13_scale
+
+    # Scale's last dim tracks intermediate / block_size (MXFP8 block_size = 32)
+    scale_block_k = intermediate // w2_scale.shape[2] if w2_scale.shape[2] > 0 else 32
+    padded_w2_scale = w2_scale.new_zeros(
+        (num_experts, hidden_size, padded_intermediate // scale_block_k)
+    )
+    padded_w2_scale[:, :, : w2_scale.shape[2]] = w2_scale
+
+    return padded_w13, padded_w13_scale, padded_w2, padded_w2_scale, padded_intermediate
+
+
+def align_mxfp8_moe_weights_for_flashinfer_trtllm(layer: Module) -> None:
+    """Prepare MXFP8 MoE weights/scales for FlashInfer TRT-LLM kernels."""
+    from flashinfer import block_scale_interleave
+    from flashinfer.fused_moe.core import (
+        get_reorder_rows_for_gated_act_gemm_row_indices,
+    )
+    from flashinfer.utils import (
+        get_shuffle_matrix_a_row_indices,
+        get_shuffle_matrix_sf_a_row_indices,
+    )
+
+    is_gated = _is_gated(layer)
+
+    w13_weight = cast(torch.Tensor, layer.w13_weight).contiguous()
+    w2_weight = cast(torch.Tensor, layer.w2_weight).contiguous()
+    w13_scale = cast(torch.Tensor, layer.w13_weight_scale_inv).contiguous()
+    w2_scale = cast(torch.Tensor, layer.w2_weight_scale_inv).contiguous()
+
+    assert w13_scale.dtype == torch.uint8
+    assert w2_scale.dtype == torch.uint8
+
+    # Pad for kernel alignment (non-gated needs 128, gated needs 16)
+    min_alignment = 16 if is_gated else 128
+    w13_weight, w13_scale, w2_weight, w2_scale, _ = _align_mxfp8_moe_weights(
+        w13_weight, w13_scale, w2_weight, w2_scale, is_gated, min_alignment
+    )
+
+    num_experts, gate_up_dim, _ = w13_weight.shape
+    _, hidden_size, _ = w2_weight.shape
+    epilogue_tile_m = 128
+
+    # Reuse precomputed row-index transforms whenever shape/device are unchanged.
+    w13_weight_u8 = w13_weight.view(torch.uint8)
+    w2_weight_u8 = w2_weight.view(torch.uint8)
+    cache_key = (
+        gate_up_dim,
+        hidden_size,
+        w2_weight.shape[-1],
+        w13_scale.shape[-1],
+        w2_scale.shape[-1],
+        epilogue_tile_m,
+        (w13_weight.device.type, w13_weight.device.index),
+        (w2_weight.device.type, w2_weight.device.index),
+        (w13_scale.device.type, w13_scale.device.index),
+        (w2_scale.device.type, w2_scale.device.index),
+    )
+    cache = _flashinfer_trtllm_shuffle_row_indices_cache_mxfp8.get(cache_key)
+    if cache is None:
+        if is_gated:
+            reorder_row_indices = get_reorder_rows_for_gated_act_gemm_row_indices(
+                w13_weight_u8[0]
+            ).to(w13_weight.device)
+        else:
+            reorder_row_indices = torch.arange(
+                gate_up_dim, device=w13_weight.device, dtype=torch.long
+            )
+        w13_shuffle_row_indices = get_shuffle_matrix_a_row_indices(
+            w13_weight_u8[0], epilogue_tile_m
+        ).to(w13_weight.device)
+        w2_shuffle_row_indices = get_shuffle_matrix_a_row_indices(
+            w2_weight_u8[0], epilogue_tile_m
+        ).to(w2_weight.device)
+        w13_scale_shuffle_row_indices = get_shuffle_matrix_sf_a_row_indices(
+            w13_scale[0].reshape(gate_up_dim, -1), epilogue_tile_m
+        ).to(w13_scale.device)
+        w2_scale_shuffle_row_indices = get_shuffle_matrix_sf_a_row_indices(
+            w2_scale[0].reshape(hidden_size, -1), epilogue_tile_m
+        ).to(w2_scale.device)
+        cache = {
+            "reorder_row_indices": reorder_row_indices,
+            "w13_shuffle_row_indices": w13_shuffle_row_indices,
+            "w2_shuffle_row_indices": w2_shuffle_row_indices,
+            "w13_scale_shuffle_row_indices": w13_scale_shuffle_row_indices,
+            "w2_scale_shuffle_row_indices": w2_scale_shuffle_row_indices,
+        }
+        _flashinfer_trtllm_shuffle_row_indices_cache_mxfp8[cache_key] = cache
+
+    reorder_row_indices = cache["reorder_row_indices"]
+    w13_shuffle_row_indices = cache["w13_shuffle_row_indices"]
+    w2_shuffle_row_indices = cache["w2_shuffle_row_indices"]
+    w13_scale_shuffle_row_indices = cache["w13_scale_shuffle_row_indices"]
+    w2_scale_shuffle_row_indices = cache["w2_scale_shuffle_row_indices"]
+
+    w13_shuffled_u8 = torch.empty_like(w13_weight_u8)
+    w2_shuffled_u8 = torch.empty_like(w2_weight_u8)
+    w13_scale_shuffled = torch.empty_like(w13_scale)
+    w2_scale_shuffled = torch.empty_like(w2_scale)
+
+    for i in range(num_experts):
+        w13_interleaved_u8 = w13_weight_u8[i].index_select(0, reorder_row_indices)
+        w13_scale_interleaved = w13_scale[i].index_select(0, reorder_row_indices)
+
+        w13_shuffled_u8[i].copy_(
+            w13_interleaved_u8.index_select(0, w13_shuffle_row_indices)
+        )
+        w2_shuffled_u8[i].copy_(w2_weight_u8[i].index_select(0, w2_shuffle_row_indices))
+
+        w13_scale_linear = w13_scale_interleaved.reshape(gate_up_dim, -1)
+        w13_scale_shuffled[i].copy_(
+            block_scale_interleave(
+                w13_scale_linear.index_select(0, w13_scale_shuffle_row_indices)
+            ).reshape_as(w13_scale_shuffled[i])
+        )
+
+        w2_scale_linear = w2_scale[i].reshape(hidden_size, -1)
+        w2_scale_shuffled[i].copy_(
+            block_scale_interleave(
+                w2_scale_linear.index_select(0, w2_scale_shuffle_row_indices)
+            ).reshape_as(w2_scale_shuffled[i])
+        )
+
+    # Keep parameter identities stable for CUDA graph capture reuse.
+    copy_or_rebind_param(layer, "w13_weight", w13_shuffled_u8.view(torch.float8_e4m3fn))
+    copy_or_rebind_param(layer, "w2_weight", w2_shuffled_u8.view(torch.float8_e4m3fn))
+    copy_or_rebind_param(
+        layer,
+        "w13_weight_scale_inv",
+        w13_scale_shuffled.contiguous(),
+    )
+    copy_or_rebind_param(
+        layer,
+        "w2_weight_scale_inv",
+        w2_scale_shuffled.contiguous(),
+    )
+    layer.w13_weight_scale_inv.format_ue8m0 = True
+    layer.w2_weight_scale_inv.format_ue8m0 = True
+
+
+def _align_fp4_moe_weights(
+    w13: torch.Tensor,
+    w13_scale: torch.Tensor,
+    w2: torch.Tensor,
+    w2_scale: torch.Tensor,
+    is_gated: bool,
+    min_alignment: int = 16,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, int]:
+    """Pad intermediate size so FlashInfer TRTLLM FP4 kernels' alignment holds.
+
+    Returns (w13, w13_scale, w2, w2_scale, padded_intermediate).
+    """
+    num_experts, hidden_size, intermediate_packed = w2.shape
+    intermediate = intermediate_packed * 2  # FP4 packs 2 values per byte
+
+    padded_intermediate = round_up_to_multiple(intermediate, min_alignment)
+    if padded_intermediate == intermediate:
+        return w13, w13_scale, w2, w2_scale, intermediate
+
+    logger.info(
+        "FP4 MoE: padding intermediate size from %d to %d (alignment=%d)",
+        intermediate,
+        padded_intermediate,
+        min_alignment,
+    )
+
+    up_mult = 2 if is_gated else 1
+    padded_gate_up = up_mult * padded_intermediate
+
+    padded_w13 = w13.new_zeros((num_experts, padded_gate_up, w13.shape[2]))
+    padded_w13[:, : w13.shape[1], :] = w13
+
+    padded_w2 = w2.new_zeros((num_experts, hidden_size, padded_intermediate // 2))
+    padded_w2[:, :, : w2.shape[2]] = w2
+
+    padded_w13_scale = w13_scale.new_zeros(
+        (num_experts, padded_gate_up, w13_scale.shape[2])
+    )
+    padded_w13_scale[:, : w13_scale.shape[1], :] = w13_scale
+
+    padded_w2_scale = w2_scale.new_zeros(
+        (num_experts, hidden_size, padded_intermediate // 16)
+    )
+    padded_w2_scale[:, :, : w2_scale.shape[2]] = w2_scale
+
+    return padded_w13, padded_w13_scale, padded_w2, padded_w2_scale, padded_intermediate
+
+
+def align_fp4_moe_weights_for_flashinfer_trtllm(layer: Module) -> None:
+    """Prepare FP4 MoE weights/scales for FlashInfer TRT-LLM kernels.
+
+    This function handles the weight transformation needed for FP4 TRTLLM MoE:
+    - Pads intermediate dimension for kernel alignment constraints
+    - Reorders weights for gated activation GEMM
+    - Shuffles weights and scales for transposed MMA output
+    - Computes the output scale factors
+    """
+    from sglang.srt.layers.quantization.utils import (
+        prepare_static_weights_for_trtllm_fp4_moe,
+    )
+
+    w13_weight = cast(torch.Tensor, layer.w13_weight)
+    w2_weight = cast(torch.Tensor, layer.w2_weight)
+    w13_weight_scale = cast(torch.Tensor, layer.w13_weight_scale)
+    w2_weight_scale = cast(torch.Tensor, layer.w2_weight_scale)
+
+    is_gated = layer.moe_runner_config.is_gated
+    min_alignment = 16 if is_gated else 128
+
+    # Pad for kernel alignment before shuffle/reorder
+    w13_weight, w13_weight_scale, w2_weight, w2_weight_scale, intermediate_size = (
+        _align_fp4_moe_weights(
+            w13_weight,
+            w13_weight_scale,
+            w2_weight,
+            w2_weight_scale,
+            is_gated,
+            min_alignment,
+        )
+    )
+
+    (
+        gemm1_weights_fp4_shuffled,
+        gemm1_scales_fp4_shuffled,
+        gemm2_weights_fp4_shuffled,
+        gemm2_scales_fp4_shuffled,
+    ) = prepare_static_weights_for_trtllm_fp4_moe(
+        w13_weight,
+        w2_weight,
+        w13_weight_scale,
+        w2_weight_scale,
+        w2_weight.size(-2),  # hidden_size
+        intermediate_size,  # padded intermediate_size
+        w13_weight.size(0),  # num_experts
+        is_gated=is_gated,
+    )
+
+    # Set flashinfer parameters in-place
+    copy_or_rebind_param(layer, "w13_weight", gemm1_weights_fp4_shuffled.contiguous())
+    copy_or_rebind_param(layer, "w2_weight", gemm2_weights_fp4_shuffled.contiguous())
+    copy_or_rebind_param(
+        layer, "w13_weight_scale", gemm1_scales_fp4_shuffled.contiguous()
+    )
+    copy_or_rebind_param(
+        layer, "w2_weight_scale", gemm2_scales_fp4_shuffled.contiguous()
+    )
+
+    # Compute additional scaling factor needed for TRT-LLM.
+    # For gated (SwiGLU): g1_scale_c = g1_alphas * a2_gscale
+    # For non-gated (Relu2): g1_scale_c = a2_gscale (no gate dequant contribution)
+    w2_input_scale_quant = cast(torch.Tensor, layer.w2_input_scale_quant)
+    g1_alphas = cast(torch.Tensor, layer.g1_alphas)
+    if layer.moe_runner_config.is_gated:
+        g1_scale_c = (w2_input_scale_quant * g1_alphas).to(torch.float32)
+    else:
+        num_experts = g1_alphas.shape[0]
+        g1_scale_c = (
+            w2_input_scale_quant.to(torch.float32).expand(num_experts).contiguous()
+        )
+    copy_or_rebind_param(layer, "g1_scale_c", g1_scale_c)
+
+    # Update intermediate_size_per_partition to reflect any padding applied
+    layer.intermediate_size_per_partition = intermediate_size
+
+
+def get_activation_type(activation: str, is_gated: bool = True) -> int:
+    """Map SGLang activation string to FlashInfer ActivationType int value."""
+    from flashinfer.fused_moe.core import ActivationType
+
+    if is_gated:
+        _ACTIVATION_STR_TO_TYPE = {
+            "silu": ActivationType.Swiglu,
+            "gelu": ActivationType.Geglu,
+        }
+    else:
+        _ACTIVATION_STR_TO_TYPE = {
+            "silu": ActivationType.Silu,
+            "gelu": ActivationType.Gelu,
+            "relu2": ActivationType.Relu2,
+        }
+    act = _ACTIVATION_STR_TO_TYPE.get(activation)
+    if act is None:
+        raise ValueError(
+            f"Unsupported activation '{activation}' for TRTLLM MoE "
+            f"(is_gated={is_gated}). "
+            f"Expected one of {list(_ACTIVATION_STR_TO_TYPE.keys())}."
+        )
+    return act.value
+
+
+@dataclass
+class FlashInferTrtllmFp8MoeQuantInfo(MoeQuantInfo):
+    """Quantization payload consumed by FlashInfer TRT-LLM FP8 MoE kernels."""
+
+    # Weights
+    w13_weight: torch.Tensor
+    w2_weight: torch.Tensor
+
+    # Expert-parallel metadata
+    global_num_experts: int
+    local_expert_offset: int
+    local_num_experts: int
+    intermediate_size: int
+
+    routing_method_type: int
+
+    # Block-quant path
+    block_quant: bool
+    use_mxfp8: bool = False
+    weight_block_k: int | None = None
+    w13_weight_scale_inv: torch.Tensor | None = None
+    w2_weight_scale_inv: torch.Tensor | None = None
+
+    # Per-tensor path
+    w13_input_scale: torch.Tensor | None = None
+    output1_scales_scalar: torch.Tensor | None = None
+    output1_scales_gate_scalar: torch.Tensor | None = None
+    output2_scales_scalar: torch.Tensor | None = None
+    use_routing_scales_on_input: bool = False
+
+    # Activation type (None = kernel default / Swiglu)
+    activation_type: int | None = None
+
+
+def _pack_topk_for_flashinfer_routed(
+    topk_ids: torch.Tensor, topk_weights: torch.Tensor
+) -> torch.Tensor:
+    """Pack routed top-k tensors into FlashInfer's int32 format."""
+    packed_ids = topk_ids.to(torch.int32)
+    packed_weights = topk_weights.to(torch.bfloat16)
+    packed = (packed_ids << 16) | packed_weights.view(torch.int16).to(torch.int32)
+    return packed
+
+
+def fused_experts_none_to_flashinfer_trtllm_fp8(
+    dispatch_output: StandardDispatchOutput,
+    quant_info: FlashInferTrtllmFp8MoeQuantInfo,
+    runner_config: MoeRunnerConfig,
+    use_routed_topk: bool = False,
+) -> StandardCombineInput:
+    from flashinfer.fused_moe import Fp8QuantizationType
+
+    from sglang.srt.layers.moe.token_dispatcher.standard import StandardCombineInput
+    from sglang.srt.layers.moe.topk import TopKOutputChecker
+    from sglang.srt.layers.moe.utils import RoutingMethodType
+
+    _SUPPORTED_FP8_ACTIVATIONS = {"silu", "relu2"}
+    assert runner_config.activation in _SUPPORTED_FP8_ACTIVATIONS, (
+        f"Only {_SUPPORTED_FP8_ACTIVATIONS} are supported for FP8 MoE, "
+        f"got '{runner_config.activation}'."
+    )
+    assert not runner_config.no_combine, "no_combine is not supported for flashinfer."
+
+    hidden_states = dispatch_output.hidden_states
+    topk_output = dispatch_output.topk_output
+    if TopKOutputChecker.format_is_bypassed(topk_output):
+        router_logits = topk_output.router_logits
+        topk_config = topk_output.topk_config
+        correction_bias = (
+            None
+            if topk_config.correction_bias is None
+            else topk_config.correction_bias.to(hidden_states.dtype)
+        )
+    else:
+        router_logits = None
+        topk_config = None
+        correction_bias = None
+
+    routing_method_type = quant_info.routing_method_type
+    fp8_quantization_type = (
+        Fp8QuantizationType.MxFp8
+        if quant_info.use_mxfp8
+        else Fp8QuantizationType.DeepSeekFp8
+    )
+    use_shuffled_weight = quant_info.use_mxfp8
+
+    if quant_info.block_quant:
+        assert quant_info.weight_block_k is not None
+        assert quant_info.w13_weight_scale_inv is not None
+        assert quant_info.w2_weight_scale_inv is not None
+
+        if quant_info.use_mxfp8:
+            assert quant_info.weight_block_k == 32
+            from flashinfer import mxfp8_quantize
+
+            a_q, a_sf = mxfp8_quantize(hidden_states, False)
+            # FlashInfer TRT-LLM MxFP8 expects token-major activation scales:
+            # [num_tokens, hidden_size // 32] (no transpose).
+            a_sf_t = a_sf.view(torch.uint8).reshape(hidden_states.shape[0], -1)
+        else:
+            a_q, a_sf = per_token_group_quant_fp8(
+                hidden_states, quant_info.weight_block_k
+            )
+            a_sf_t = a_sf.t().contiguous()
+
+        # Allocate output inside symmetric memory context
+        with use_symmetric_memory(
+            get_tp_group(), disabled=not is_allocation_symmetric()
+        ):
+            symm_output = torch.empty(
+                hidden_states.shape[0],
+                hidden_states.shape[1],
+                dtype=hidden_states.dtype,
+                device=hidden_states.device,
+            )
+
+        # Move kernel call outside context manager to avoid graph breaks
+        # during torch.compile for piecewise cuda graph.
+        # Use custom op wrapper for torch.compile compatibility.
+        if use_routed_topk:
+            assert (
+                runner_config.top_k is not None
+            ), "runner_config.top_k is required for flashinfer_trtllm_routed."
+            assert TopKOutputChecker.format_is_standard(topk_output)
+            packed_topk_ids = _pack_topk_for_flashinfer_routed(
+                topk_ids=topk_output.topk_ids,
+                topk_weights=topk_output.topk_weights,
+            )
+
+            output = trtllm_fp8_block_scale_routed_moe_wrapper(
+                topk_ids=packed_topk_ids,
+                routing_bias=None,
+                hidden_states=a_q,
+                hidden_states_scale=a_sf_t,
+                gemm1_weights=quant_info.w13_weight,
+                gemm1_weights_scale=quant_info.w13_weight_scale_inv,
+                gemm2_weights=quant_info.w2_weight,
+                gemm2_weights_scale=quant_info.w2_weight_scale_inv,
+                num_experts=quant_info.global_num_experts,
+                top_k=runner_config.top_k,
+                n_group=None,
+                topk_group=None,
+                intermediate_size=quant_info.intermediate_size,
+                local_expert_offset=quant_info.local_expert_offset,
+                local_num_experts=quant_info.local_num_experts,
+                routed_scaling_factor=(
+                    runner_config.routed_scaling_factor
+                    if runner_config.routed_scaling_factor is not None
+                    else 1.0
+                ),
+                routing_method_type=(
+                    RoutingMethodType.TopK
+                    if routing_method_type == RoutingMethodType.DeepSeekV3
+                    else routing_method_type
+                ),
+                use_shuffled_weight=use_shuffled_weight,
+                tune_max_num_tokens=next_power_of_2(a_q.shape[0]),
+                fp8_quantization_type=int(fp8_quantization_type),
+                activation_type=quant_info.activation_type,
+            )
+        else:
+            assert TopKOutputChecker.format_is_bypassed(topk_output)
+
+            output = trtllm_fp8_block_scale_moe_wrapper(
+                routing_logits=router_logits,
+                routing_bias=correction_bias,
+                hidden_states=a_q,
+                hidden_states_scale=a_sf_t,
+                gemm1_weights=quant_info.w13_weight,
+                gemm1_weights_scale=quant_info.w13_weight_scale_inv,
+                gemm2_weights=quant_info.w2_weight,
+                gemm2_weights_scale=quant_info.w2_weight_scale_inv,
+                num_experts=quant_info.global_num_experts,
+                top_k=topk_config.top_k,
+                n_group=topk_config.num_expert_group,
+                topk_group=topk_config.topk_group,
+                intermediate_size=quant_info.intermediate_size,
+                local_expert_offset=quant_info.local_expert_offset,
+                local_num_experts=quant_info.local_num_experts,
+                routed_scaling_factor=(
+                    runner_config.routed_scaling_factor
+                    if runner_config.routed_scaling_factor is not None
+                    else 1.0
+                ),
+                routing_method_type=routing_method_type,
+                use_shuffled_weight=use_shuffled_weight,
+                tune_max_num_tokens=next_power_of_2(a_q.shape[0]),
+                fp8_quantization_type=int(fp8_quantization_type),
+                activation_type=quant_info.activation_type,
+            )
+        # TODO: Once https://github.com/flashinfer-ai/flashinfer/issues/2703 is fixed, pass output to moe kernel and remove this copy.
+        symm_output.copy_(output)
+        output = symm_output
+    else:
+        assert TopKOutputChecker.format_is_bypassed(topk_output)
+        assert quant_info.w13_input_scale is not None
+        assert quant_info.output1_scales_scalar is not None
+        assert quant_info.output1_scales_gate_scalar is not None
+        assert quant_info.output2_scales_scalar is not None
+
+        a_q, _ = scaled_fp8_quant(hidden_states, quant_info.w13_input_scale)
+        routing_bias_cast = (
+            None if correction_bias is None else correction_bias.to(torch.bfloat16)
+        )
+
+        # Allocate output inside symmetric memory context
+        with use_symmetric_memory(
+            get_tp_group(), disabled=not is_allocation_symmetric()
+        ):
+            symm_output = torch.empty(
+                hidden_states.shape[0],
+                hidden_states.shape[1],
+                dtype=torch.bfloat16,
+                device=hidden_states.device,
+            )
+
+        # Move kernel call outside context manager to avoid graph breaks
+        # during torch.compile for piecewise cuda graph.
+        # Use custom op wrapper for torch.compile compatibility.
+
+        router_logits = router_logits.to(torch.bfloat16)
+
+        output = trtllm_fp8_per_tensor_scale_moe_wrapper(
+            routing_logits=router_logits,
+            routing_bias=routing_bias_cast,
+            hidden_states=a_q,
+            gemm1_weights=quant_info.w13_weight,
+            output1_scales_scalar=quant_info.output1_scales_scalar,
+            output1_scales_gate_scalar=quant_info.output1_scales_gate_scalar,
+            gemm2_weights=quant_info.w2_weight,
+            output2_scales_scalar=quant_info.output2_scales_scalar,
+            num_experts=quant_info.global_num_experts,
+            top_k=topk_config.top_k,
+            n_group=topk_config.num_expert_group,
+            topk_group=topk_config.topk_group,
+            intermediate_size=int(quant_info.w2_weight.shape[2]),
+            local_expert_offset=quant_info.local_expert_offset,
+            local_num_experts=quant_info.local_num_experts,
+            routed_scaling_factor=(
+                runner_config.routed_scaling_factor
+                if runner_config.routed_scaling_factor is not None
+                else 1.0
+            ),
+            use_routing_scales_on_input=False,
+            routing_method_type=routing_method_type,
+            tune_max_num_tokens=next_power_of_2(a_q.shape[0]),
+            activation_type=quant_info.activation_type,
+        )
+        symm_output.copy_(output)
+        output = symm_output
+
+    return StandardCombineInput(hidden_states=output)
+
+
+@dataclass
+class FlashInferTrtllmFp4MoeQuantInfo(MoeQuantInfo):
+    """Quantization payload consumed by FlashInfer TRT-LLM FP4 MoE kernels."""
+
+    w13_weight: torch.Tensor
+    w2_weight: torch.Tensor
+    w13_weight_scale: torch.Tensor
+    w2_weight_scale: torch.Tensor
+
+    # Scaling factors
+    g1_scale_c: torch.Tensor
+    g1_alphas: torch.Tensor
+    g2_alphas: torch.Tensor
+    w13_input_scale_quant: torch.Tensor
+
+    # Expert-parallel metadata
+    global_num_experts: int
+    local_expert_offset: int
+    local_num_experts: int
+    intermediate_size_per_partition: int
+
+    routing_method_type: int
+    use_per_token_activation: bool = False
+
+
+def quantize_hidden_states_fp4(
+    hidden_states: torch.Tensor,
+    input_scale_quant: torch.Tensor,
+) -> tuple[torch.Tensor, torch.Tensor]:
+    """
+    Quantize hidden states to FP4 for TRTLLM MoE.
+
+    Global scale factor is set by ModelOptNvFp4FusedMoEMethod during weight loading.
+    Only block scales are computed at runtime for efficiency.
+
+    Returns (packed_fp4_uint8, scale_float8_e4m3fn_runtime)
+    """
+
+    # flashinfer.fp4_quantize returns (packed_uint8, scale_fp8)
+    # Only the block scales are computed at runtime
+    hs_fp4_bytes, hs_sf_bytes = fp4_quantize(
+        hidden_states,
+        input_scale_quant,
+        16,  # sf_vec_size
+        False,  # use_ue8m0
+        False,  # is_sf_swizzled_layout
+    )
+
+    seq_len, hidden_size = hidden_states.shape
+    hs_fp4 = hs_fp4_bytes.reshape(seq_len, hidden_size // 2)
+    # TRT-LLM expects hidden state scales shaped as [seq_len, hidden_size // 16]
+    hs_sf = hs_sf_bytes.view(torch.float8_e4m3fn).reshape(seq_len, hidden_size // 16)
+
+    return hs_fp4, hs_sf
+
+
+def fused_experts_none_to_flashinfer_trtllm_fp4(
+    dispatch_output: StandardDispatchOutput,
+    quant_info: FlashInferTrtllmFp4MoeQuantInfo,
+    runner_config: MoeRunnerConfig,
+    use_routed_topk: bool = False,
+) -> StandardCombineInput:
+    """FlashInfer TRTLLM FP4 MoE forward pass.
+
+    This function handles the FP4 TRTLLM MoE path that was previously in
+    ModelOptNvFp4FusedMoEMethod.apply.
+    """
+    from flashinfer.fused_moe import (
+        trtllm_fp4_block_scale_moe,
+        trtllm_fp4_block_scale_routed_moe,
+    )
+
+    from sglang.srt.layers.moe.token_dispatcher.standard import StandardCombineInput
+    from sglang.srt.layers.moe.topk import TopKOutputChecker
+    from sglang.srt.layers.moe.utils import RoutingMethodType
+
+    _SUPPORTED_FP4_ACTIVATIONS = {"silu", "relu2", "gelu"}
+    assert runner_config.activation in _SUPPORTED_FP4_ACTIVATIONS, (
+        f"Only {_SUPPORTED_FP4_ACTIVATIONS} are supported for FP4 MoE, "
+        f"got '{runner_config.activation}'."
+    )
+
+    hidden_states = dispatch_output.hidden_states
+    topk_output = dispatch_output.topk_output
+
+    # Quantize hidden states to FP4
+    if quant_info.use_per_token_activation:
+        from flashinfer import SfLayout, nvfp4_quantize
+
+        e4m3_max = 448.0
+        if (
+            envs.FLASHINFER_NVFP4_4OVER6.get()
+            and envs.FLASHINFER_NVFP4_4OVER6_E4M3_USE_256.get()
+        ):
+            e4m3_max = 256.0
+
+        hs_fp4_bytes, hs_sf_bytes, per_token_scale = nvfp4_quantize(
+            hidden_states,
+            1.0 / (e4m3_max * 6.0),
+            sfLayout=SfLayout.layout_linear,
+            per_token_activation=True,
+        )
+
+        seq_len, hidden_size = hidden_states.shape
+        hs_fp4 = hs_fp4_bytes.reshape(seq_len, hidden_size // 2)
+        hs_scale_linear = hs_sf_bytes.view(torch.float8_e4m3fn).reshape(
+            seq_len, hidden_size // 16
+        )
+    else:
+        per_token_scale = None
+        hs_fp4, hs_scale_linear = quantize_hidden_states_fp4(
+            hidden_states, quant_info.w13_input_scale_quant
+        )
+    hs_scale = hs_scale_linear.view(torch.float8_e4m3fn).reshape(
+        *hs_scale_linear.shape[:-1], -1
+    )
+    activation_type = get_activation_type(
+        runner_config.activation, is_gated=runner_config.is_gated
+    )
+
+    # Build per-expert clamp-limit tensor from the per-layer scalar.
+    _clamp_val = runner_config.gemm1_clamp_limit
+    if _clamp_val is not None:
+        gemm1_clamp_limit = torch.full(
+            (quant_info.local_num_experts,),
+            _clamp_val,
+            dtype=torch.float32,
+            device=hs_fp4.device,
+        )
+    else:
+        gemm1_clamp_limit = None
+
+    # The TRTLLM FP4 kernels cannot run a parameterized swiglu: the
+    # gemm1_alpha/gemm1_beta tensors are ignored under ActivationType.Swiglu
+    # (verified bit-identical outputs with and without them), and
+    # ActivationType.SwigluBias is rejected by activationTypeToGatedActType.
+    # Fail fast instead of silently generating garbage.
+    if runner_config.gemm1_alpha is not None:
+        raise NotImplementedError(
+            "flashinfer_trtllm FP4 MoE does not support parameterized "
+            "(GPT-OSS-style) SwiGLU (gemm1_alpha is set); use "
+            "--moe-runner-backend flashinfer_cutlass instead."
+        )
+
+    num_tokens = hs_fp4.shape[0]
+    hidden_size = (
+        hs_fp4.shape[-1] * 2 if hs_fp4.dtype == torch.uint8 else hs_fp4.shape[-1]
+    )
+    _provided = _moe_output_buf.get()
+    _symm_required = is_allocation_symmetric()
+    if (
+        _provided is not None
+        and _provided.shape == (num_tokens, hidden_size)
+        and _provided.dtype == hidden_states.dtype
+        and _provided.device == hs_fp4.device
+        and (
+            not _symm_required
+            or not is_symmetric_memory_enabled()
+            or is_tensor_in_symmetric_mempool(_provided)
+        )
+    ):
+        symm_output = _provided
+    else:
+        with use_symmetric_memory(get_tp_group(), disabled=not _symm_required):
+            symm_output = torch.empty(
+                num_tokens, hidden_size, dtype=hidden_states.dtype, device=hs_fp4.device
+            )
+
+    # Fall back to routed path when topk was already materialized (e.g. sigmoid routing).
+    if not use_routed_topk and TopKOutputChecker.format_is_standard(topk_output):
+        use_routed_topk = True
+
+    if use_routed_topk:
+        assert TopKOutputChecker.format_is_standard(topk_output)
+
+        packed_topk_ids = _pack_topk_for_flashinfer_routed(
+            topk_output.topk_ids, topk_output.topk_weights
+        )
+        result = trtllm_fp4_block_scale_routed_moe(
+            topk_ids=packed_topk_ids,
+            routing_bias=None,
+            hidden_states=hs_fp4,
+            hidden_states_scale=hs_scale,
+            gemm1_weights=quant_info.w13_weight,
+            gemm1_weights_scale=quant_info.w13_weight_scale.view(torch.float8_e4m3fn),
+            gemm1_bias=None,
+            gemm1_alpha=None,
+            gemm1_beta=None,
+            gemm1_clamp_limit=gemm1_clamp_limit,
+            gemm2_weights=quant_info.w2_weight,
+            gemm2_weights_scale=quant_info.w2_weight_scale.view(torch.float8_e4m3fn),
+            gemm2_bias=None,
+            output1_scale_scalar=quant_info.g1_scale_c,
+            output1_scale_gate_scalar=quant_info.g1_alphas,
+            output2_scale_scalar=quant_info.g2_alphas,
+            per_token_scale=per_token_scale,
+            num_experts=quant_info.global_num_experts,
+            top_k=topk_output.topk_ids.shape[1],
+            n_group=0,
+            topk_group=0,
+            intermediate_size=quant_info.intermediate_size_per_partition,
+            local_expert_offset=quant_info.local_expert_offset,
+            local_num_experts=quant_info.local_num_experts,
+            routed_scaling_factor=None,
+            routing_method_type=1,  # Unused, but must be 1 to pass validation.
+            do_finalize=True,
+            activation_type=activation_type,
+            tune_max_num_tokens=next_power_of_2(hs_fp4.shape[0]),
+            output=symm_output,
+        )[0]
+    else:
+        assert TopKOutputChecker.format_is_bypassed(topk_output)
+
+        router_logits = topk_output.router_logits
+        topk_config = topk_output.topk_config
+        routing_method_type = quant_info.routing_method_type
+
+        correction_bias = (
+            None
+            if topk_config.correction_bias is None
+            else topk_config.correction_bias.to(hidden_states.dtype)
+        )
+        result = trtllm_fp4_block_scale_moe(
+            routing_logits=router_logits,
+            routing_bias=correction_bias,
+            hidden_states=hs_fp4,
+            hidden_states_scale=hs_scale,
+            gemm1_weights=quant_info.w13_weight,
+            gemm1_weights_scale=quant_info.w13_weight_scale.view(torch.float8_e4m3fn),
+            gemm1_bias=None,
+            gemm1_alpha=None,
+            gemm1_beta=None,
+            gemm1_clamp_limit=gemm1_clamp_limit,
+            gemm2_weights=quant_info.w2_weight,
+            gemm2_weights_scale=quant_info.w2_weight_scale.view(torch.float8_e4m3fn),
+            gemm2_bias=None,
+            output1_scale_scalar=quant_info.g1_scale_c,
+            output1_scale_gate_scalar=quant_info.g1_alphas,
+            output2_scale_scalar=quant_info.g2_alphas,
+            per_token_scale=per_token_scale,
+            num_experts=quant_info.global_num_experts,
+            top_k=topk_config.top_k,
+            n_group=topk_config.num_expert_group,
+            topk_group=topk_config.topk_group,
+            intermediate_size=quant_info.intermediate_size_per_partition,
+            local_expert_offset=quant_info.local_expert_offset,
+            local_num_experts=quant_info.local_num_experts,
+            routed_scaling_factor=runner_config.routed_scaling_factor,
+            routing_method_type=(
+                routing_method_type
+                if routing_method_type is not None
+                else RoutingMethodType.Default
+            ),
+            do_finalize=True,
+            activation_type=activation_type,
+            tune_max_num_tokens=next_power_of_2(hs_fp4.shape[0]),
+            output=symm_output,
+        )[0]
+
+    return StandardCombineInput(hidden_states=result)
+
+
+@dataclass
+class FlashInferTrtllmBf16MoeQuantInfo(MoeQuantInfo):
+    """Quantization payload consumed by FlashInfer TRT-LLM BF16 MoE kernels."""
+
+    gemm1_weights: torch.Tensor
+    gemm2_weights: torch.Tensor
+
+    # Expert-parallel metadata
+    global_num_experts: int
+    local_expert_offset: int
+
+
+def fused_experts_none_to_flashinfer_trtllm_bf16(
+    dispatch_output: StandardDispatchOutput,
+    quant_info: FlashInferTrtllmBf16MoeQuantInfo,
+    runner_config: MoeRunnerConfig,
+    use_routed_topk: bool = False,
+) -> StandardCombineInput:
+    # lazy import
+    from sglang.srt.layers.moe.token_dispatcher.standard import StandardCombineInput
+    from sglang.srt.layers.moe.topk import TopKOutputChecker
+    from sglang.srt.layers.moe.utils import RoutingMethodType
+
+    trtllm_bf16_routed_moe = None
+    trtllm_bf16_moe = None
+    if use_routed_topk:
+        try:
+            from flashinfer.fused_moe import trtllm_bf16_routed_moe
+        except ImportError as e:
+            raise ImportError(
+                "Can't import trtllm_bf16_routed_moe from flashinfer. "
+                "Please check flashinfer version to use bf16 with flashinfer_trtllm_routed backend."
+            ) from e
+    else:
+        try:
+            from flashinfer.fused_moe import trtllm_bf16_moe
+        except ImportError as e:
+            raise ImportError(
+                "Can't import trtllm_bf16_moe from flashinfer. "
+                "Please check flashinfer version to use bf16 with flashinfer_trtllm backend."
+            ) from e
+
+    _SUPPORTED_BF16_ACTIVATIONS = {"silu", "relu2"}
+    assert runner_config.activation in _SUPPORTED_BF16_ACTIVATIONS, (
+        f"Only {_SUPPORTED_BF16_ACTIVATIONS} are supported for flashinfer trtllm bf16 moe, "
+        f"got '{runner_config.activation}'."
+    )
+    if not use_routed_topk:
+        assert (
+            dispatch_output.topk_output.topk_config.renormalize
+        ), "Renormalize is required for flashinfer trtllm moe"
+    assert (
+        runner_config.num_fused_shared_experts == 0
+    ), "Fused shared experts are not supported for flashinfer trtllm moe"
+    activation_type = get_activation_type(
+        runner_config.activation, is_gated=runner_config.is_gated
+    )
+
+    hidden_states = dispatch_output.hidden_states
+    topk_output = dispatch_output.topk_output
+
+    with use_symmetric_memory(get_tp_group(), disabled=not is_allocation_symmetric()):
+        if use_routed_topk:
+            assert (
+                runner_config.top_k is not None
+            ), "runner_config.top_k is required for flashinfer_trtllm_routed."
+            assert TopKOutputChecker.format_is_standard(topk_output)
+            routing_method_type = runner_config.routing_method_type
+            if routing_method_type is None:
+                routing_method_type = RoutingMethodType.Default
+            elif routing_method_type == RoutingMethodType.DeepSeekV3:
+                routing_method_type = RoutingMethodType.TopK
+
+            packed_topk_ids = _pack_topk_for_flashinfer_routed(
+                topk_ids=topk_output.topk_ids,
+                topk_weights=topk_output.topk_weights,
+            )
+            final_hidden_states = trtllm_bf16_routed_moe(
+                topk_ids=packed_topk_ids,
+                hidden_states=hidden_states,
+                gemm1_weights=quant_info.gemm1_weights,
+                gemm2_weights=quant_info.gemm2_weights,
+                num_experts=quant_info.global_num_experts,
+                top_k=runner_config.top_k,
+                n_group=None,
+                topk_group=None,
+                intermediate_size=runner_config.intermediate_size_per_partition,
+                local_expert_offset=quant_info.local_expert_offset,
+                local_num_experts=runner_config.num_local_experts,
+                routing_method_type=routing_method_type,
+                routed_scaling_factor=(
+                    runner_config.routed_scaling_factor
+                    if runner_config.routed_scaling_factor is not None
+                    else 1.0
+                ),
+                tune_max_num_tokens=next_power_of_2(hidden_states.shape[0]),
+                activation_type=activation_type,
+            )
+        else:
+            assert TopKOutputChecker.format_is_bypassed(topk_output)
+            topk_config = topk_output.topk_config
+
+            # Call the fused kernel
+            final_hidden_states = trtllm_bf16_moe(
+                routing_logits=topk_output.router_logits,
+                routing_bias=topk_config.correction_bias,
+                hidden_states=hidden_states,
+                gemm1_weights=quant_info.gemm1_weights,
+                gemm2_weights=quant_info.gemm2_weights,
+                num_experts=quant_info.global_num_experts,
+                top_k=topk_config.top_k,
+                n_group=topk_config.num_expert_group,
+                topk_group=topk_config.topk_group,
+                intermediate_size=runner_config.intermediate_size_per_partition,
+                local_expert_offset=quant_info.local_expert_offset,
+                local_num_experts=runner_config.num_local_experts,
+                routing_method_type=runner_config.routing_method_type,
+                routed_scaling_factor=runner_config.routed_scaling_factor,
+                tune_max_num_tokens=next_power_of_2(hidden_states.shape[0]),
+                activation_type=activation_type,
+            )
+
+    return StandardCombineInput(hidden_states=final_hidden_states)
+
+
+@register_fused_func("none", "flashinfer_trtllm")
+def fused_experts_none_to_flashinfer_trtllm(
+    dispatch_output: StandardDispatchOutput,
+    quant_info: MoeQuantInfo,
+    runner_config: MoeRunnerConfig,
+) -> StandardCombineInput:
+    """Dispatch to FP8 or FP4 FlashInfer TRT-LLM MoE based on quant_info type."""
+    if isinstance(quant_info, FlashInferTrtllmFp4MoeQuantInfo):
+        return fused_experts_none_to_flashinfer_trtllm_fp4(
+            dispatch_output, quant_info, runner_config
+        )
+    if isinstance(quant_info, FlashInferTrtllmFp8MoeQuantInfo):
+        return fused_experts_none_to_flashinfer_trtllm_fp8(
+            dispatch_output, quant_info, runner_config
+        )
+    if isinstance(quant_info, FlashInferTrtllmBf16MoeQuantInfo):
+        return fused_experts_none_to_flashinfer_trtllm_bf16(
+            dispatch_output, quant_info, runner_config
+        )
+    raise TypeError(
+        f"Unexpected quant_info type for flashinfer_trtllm: {type(quant_info)}"
+    )
+
+
+@register_fused_func("none", "flashinfer_trtllm_routed")
+def fused_experts_none_to_flashinfer_trtllm_routed(
+    dispatch_output: StandardDispatchOutput,
+    quant_info: MoeQuantInfo,
+    runner_config: MoeRunnerConfig,
+) -> StandardCombineInput:
+    if isinstance(quant_info, FlashInferTrtllmFp4MoeQuantInfo):
+        return fused_experts_none_to_flashinfer_trtllm_fp4(
+            dispatch_output,
+            quant_info,
+            runner_config,
+            use_routed_topk=True,
+        )
+    if isinstance(quant_info, FlashInferTrtllmFp8MoeQuantInfo):
+        return fused_experts_none_to_flashinfer_trtllm_fp8(
+            dispatch_output,
+            quant_info,
+            runner_config,
+            use_routed_topk=True,
+        )
+    if isinstance(quant_info, FlashInferTrtllmBf16MoeQuantInfo):
+        return fused_experts_none_to_flashinfer_trtllm_bf16(
+            dispatch_output,
+            quant_info,
+            runner_config,
+            use_routed_topk=True,
+        )
+    raise TypeError(
+        f"Unexpected quant_info type for flashinfer_trtllm_routed: {type(quant_info)}"
+    )
+
+
+# Register the experimental experimental_sgl_trtllm MoE fused-func (MoeRunner needs it at
+# build time even for LoRA); gated by the master switch so the upstream path is untouched.
+if _SGLANG_EXPERIMENTAL_LORA_OPTI:
+    from sglang.srt.lora.trtllm_lora_temp import sgl_backend  # noqa: E402,F401

special_tokens_map.json

@@ -0,0 +1,16 @@
+{
+  "bos_token": {
+    "content": "]~b]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "[e~[",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,501 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "200000": {
+      "content": "]!p~[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200001": {
+      "content": "<fim_prefix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200002": {
+      "content": "<fim_middle>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200003": {
+      "content": "<fim_suffix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200004": {
+      "content": "<fim_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200005": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200006": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200007": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200008": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200009": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200010": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200011": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200012": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200013": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200014": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200015": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200016": {
+      "content": "<commit_before>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200017": {
+      "content": "<commit_msg>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200018": {
+      "content": "<commit_after>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200019": {
+      "content": "]~b]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200020": {
+      "content": "[e~[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200021": {
+      "content": "]!d~[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200022": {
+      "content": "<function_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200023": {
+      "content": "<code_interpreter>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200024": {
+        "content": "]<]speech[>[",
+        "lstrip": false,
+        "normalized": false,
+        "rstrip": false,
+        "single_word": false,
+        "special": true
+    },
+    "200025": {
+      "content": "]<]image[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200026": {
+      "content": "]<]video[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200027": {
+      "content": "]<]start of speech[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200028": {
+      "content": "]<]end of speech[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200029": {
+      "content": "]<]start of image[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200030": {
+      "content": "]<]end of image[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200031": {
+      "content": "]<]start of video[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200032": {
+      "content": "]<]end of video[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200033": {
+      "content": "]<]vision pad[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200034": {
+      "content": "]~!b[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200035": {
+      "content": "<jupyter_error>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200036": {
+      "content": "<add_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200037": {
+      "content": "<delete_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200038": {
+      "content": "<rename_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200039": {
+      "content": "<edit_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200040": {
+      "content": "<commit_message>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200041": {
+      "content": "<empty_source_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200042": {
+      "content": "<repo_struct>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200043": {
+      "content": "<code_context>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200044": {
+      "content": "<file_content>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200045": {
+      "content": "<source_files>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200046": {
+      "content": "<pr_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200047": {
+      "content": "<review_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200048": {
+      "content": "<filepath>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200049": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200050": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200051": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200052": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200053": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200054": {
+      "content": "]<]frame[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200055": {
+      "content": "]<]start of frame[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200056": {
+      "content": "]<]end of frame[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200057": {
+      "content": "<|content_altered_placeholder|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200058": {
+      "content": "]<]minimax[>[",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200059": {
+      "content": "<mm:think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200060": {
+      "content": "</mm:think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "]~b]",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "[e~[",
+  "pad_token": "]!p~[",
+  "model_max_length": 40960000,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[e~["
+}
+

video_processor.py

@@ -0,0 +1,208 @@
+# Copyright 2023-2024 SGLang Team
+# Licensed under the Apache License, Version 2.0 (the "License");
+"""
+MiniMax VL family HuggingFace-compatible VideoProcessor.
+"""
+
+import math
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torchvision
+from torchvision.transforms import InterpolationMode
+from transformers import BatchFeature
+from transformers.image_utils import PILImageResampling, SizeDict
+from transformers.processing_utils import (
+    Unpack,
+    VideosKwargs,
+)
+from transformers.utils import TensorType
+from transformers.video_processing_utils import BaseVideoProcessor
+from transformers.video_utils import group_videos_by_shape, reorder_videos
+
+MAX_RATIO = 200
+
+
+def round_by_factor(number: int, factor: int) -> int:
+    return round(number / factor) * factor
+
+
+def ceil_by_factor(number: int, factor: int) -> int:
+    return math.ceil(number / factor) * factor
+
+
+def floor_by_factor(number: int, factor: int) -> int:
+    return math.floor(number / factor) * factor
+
+
+def smart_resize(
+    height: int,
+    width: int,
+    factor: int = 28,
+    min_pixels: int = 4 * 28 * 28,
+    max_pixels: int = 451584,
+) -> tuple[int, int]:
+    if max(height, width) / min(height, width) > MAX_RATIO:
+        raise ValueError(
+            f"absolute aspect ratio must be smaller than {MAX_RATIO}, "
+            f"got {max(height, width) / min(height, width)}"
+        )
+    h_bar = max(factor, round_by_factor(height, factor))
+    w_bar = max(factor, round_by_factor(width, factor))
+    if h_bar * w_bar > max_pixels:
+        beta = math.sqrt((height * width) / max_pixels)
+        h_bar = floor_by_factor(height / beta, factor)
+        w_bar = floor_by_factor(width / beta, factor)
+    elif h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (height * width))
+        h_bar = ceil_by_factor(height * beta, factor)
+        w_bar = ceil_by_factor(width * beta, factor)
+    return h_bar, w_bar
+
+
+class MiniMaxM3VLVideoProcessorKwargs(VideosKwargs, total=False):
+    patch_size: int
+    temporal_patch_size: int
+    merge_size: int
+    min_pixels: int
+    max_pixels: int
+    total_pixels: int
+    min_frames: int
+    max_frames: int
+    fps: float | int
+
+
+class MiniMaxM3VLVideoProcessor(BaseVideoProcessor):
+    do_resize = True
+    resample = PILImageResampling.BICUBIC
+    size = {"height": 672, "width": 672}
+    default_to_square = False
+    do_rescale = True
+    rescale_factor = 1 / 255
+    do_normalize = True
+    image_mean = [0.48145466, 0.4578275, 0.40821073]
+    image_std = [0.26862954, 0.26130258, 0.27577711]
+    do_convert_rgb = True
+    do_sample_frames = False
+    patch_size = 14
+    temporal_patch_size = 2
+    merge_size = 2
+    min_pixels = 4 * 28 * 28
+    max_pixels = 768 * 28 * 28                  # 602,112
+    total_pixels = int(64000 * 28 * 28 * 0.9)   # ~45M, ~64k tokens budget
+    fps = 1.0
+    min_frames = 4
+    max_frames = 768
+    valid_kwargs = MiniMaxM3VLVideoProcessorKwargs
+    model_input_names = ["pixel_values_videos", "video_grid_thw"]
+
+    def __init__(self, **kwargs: Unpack[MiniMaxM3VLVideoProcessorKwargs]):
+        super().__init__(**kwargs)
+
+    def _preprocess(
+        self,
+        videos: List[torch.Tensor],
+        do_convert_rgb: bool,
+        do_resize: bool,
+        size: SizeDict,
+        resample: PILImageResampling | InterpolationMode | int | None,
+        do_rescale: bool,
+        rescale_factor: float,
+        do_normalize: bool,
+        image_mean: float | List[float] | None,
+        image_std: float | List[float] | None,
+        patch_size: int,
+        temporal_patch_size: int,
+        merge_size: int,
+        min_pixels: int,
+        max_pixels: int,
+        return_tensors: str | TensorType | None = None,
+        **kwargs,
+    ) -> BatchFeature:
+        grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
+        resized_videos_grouped = {}
+        factor = patch_size * merge_size
+        for shape, stacked_videos in grouped_videos.items():
+            batch_size, num_frames, channels, height, width = stacked_videos.shape
+            resized_height, resized_width = height, width
+            if do_resize:
+                resized_height, resized_width = smart_resize(
+                    height, width, factor=factor,
+                    min_pixels=min_pixels, max_pixels=max_pixels,
+                )
+                stacked_videos = stacked_videos.view(
+                    batch_size * num_frames, channels, height, width
+                )
+                stacked_videos = self.resize(
+                    stacked_videos,
+                    size=SizeDict(height=resized_height, width=resized_width),
+                    resample=resample,
+                )
+                stacked_videos = stacked_videos.view(
+                    batch_size,
+                    num_frames,
+                    channels,
+                    resized_height,
+                    resized_width,
+                )
+            resized_videos_grouped[shape] = stacked_videos
+        resized_videos = reorder_videos(resized_videos_grouped, grouped_videos_index)
+
+        grouped_videos, grouped_videos_index = group_videos_by_shape(resized_videos)
+        processed_videos_grouped = {}
+        processed_grids = {}
+        for shape, stacked_videos in grouped_videos.items():
+            resized_height, resized_width = stacked_videos.shape[-2:]
+            patches = self.rescale_and_normalize(
+                stacked_videos,
+                do_rescale,
+                rescale_factor,
+                do_normalize,
+                image_mean,
+                image_std,
+            )
+
+            if pad := -patches.shape[1] % temporal_patch_size:
+                repeats = patches[:, -1:].expand(-1, pad, -1, -1, -1)
+                patches = torch.cat([patches, repeats], dim=1)
+
+            batch_size, grid_t, channels = patches.shape[:3]
+            grid_t = grid_t // temporal_patch_size
+            grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+
+            patches = patches.view(
+                batch_size,
+                grid_t,
+                temporal_patch_size,
+                channels,
+                grid_h // merge_size,
+                merge_size,
+                patch_size,
+                grid_w // merge_size,
+                merge_size,
+                patch_size,
+            )
+            patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
+            flatten_patches = patches.reshape(
+                batch_size,
+                grid_t * grid_h * grid_w,
+                channels * temporal_patch_size * patch_size * patch_size,
+            )
+
+            processed_videos_grouped[shape] = flatten_patches
+            processed_grids[shape] = [[grid_t, grid_h, grid_w]] * batch_size
+
+        processed_videos = reorder_videos(
+            processed_videos_grouped, grouped_videos_index
+        )
+        processed_grids = reorder_videos(processed_grids, grouped_videos_index)
+        pixel_values_videos = torch.cat(processed_videos, dim=0)
+        video_grid_thw = torch.tensor(processed_grids, dtype=torch.long)
+
+        return BatchFeature(
+            data={
+                "pixel_values_videos": pixel_values_videos,
+                "video_grid_thw": video_grid_thw,
+            },
+            tensor_type=return_tensors,
+        )