Bug Report: Memory Allocation Error in Qwen2.5-7B/Qwen3-4B Conversion to QNN-Supported ONNX

[nnbw-liu]

Describe the bug
The following error occurs when converting the Qwen model to a QNN-supported ONNX model:

2025-10-11 20:19:31.4721216 [E:onnxruntime:, [sequential_executor.cc:572](http://sequential_executor.cc:572/) onnxruntime::ExecuteKernel] Non-zero status code returned while running Mul node. Name:'/model/layers.14/mlp/act_fn/Mul' Status Message: E:_work\1\s\onnxruntime\core\framework\[bfc_arena.cc:359](http://bfc_arena.cc:359/) onnxruntime::BFCArena::AllocateRawInternal Failed to allocate memory for requested buffer of size 159383552

Tested models: Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507​

To Reproduce
Configure the environment and run according to the guidelines in olive-recipes/Qwen-Qwen2.5-7B-Instruct/QNN at main · microsoft/olive-recipes

Expected behavior
Output a QNN-supported ONNX model​

Olive config

{
    "input_model": { "type": "HfModel", "model_path": "Qwen/Qwen3-4B-Instruct-2507" },
    "systems": {
        "qnn_system": {
            "type": "PythonEnvironment",
            "python_environment_path": "F:\\AI\\gaotong\\olive\\qnn\\.venv\\Scripts",
            "accelerators": [ { "execution_providers": [ "QNNExecutionProvider" ] } ]
        }
    },
    "data_configs": [
        {
            "name": "wikitext2_train_joined",
            "type": "HuggingfaceContainer",
            "load_dataset_config": { "data_name": "wikitext", "subset": "wikitext-2-raw-v1", "split": "train" },
            "pre_process_data_config": {
                "strategy": "join",
                "add_special_tokens": false,
                "max_seq_len": 4096,
                "max_samples": 128
            }
        },
        {
            "name": "wikitext2_train_act",
            "type": "HuggingfaceContainer",
            "load_dataset_config": { "data_name": "wikitext", "subset": "wikitext-2-raw-v1", "split": "train" },
            "pre_process_data_config": {
                "strategy": "line-by-line",
                "add_special_tokens": true,
                "max_samples": 256,
                "max_seq_len": 4096
            }
        }
    ],
    "passes": {
        "q": { "type": "QuaRot" },
        "g": {
            "type": "GptqModel",
            "bits": 4,
            "sym": true,
            "group_size": -1,
            "lm_head": false,
            "device": "cuda",
            "data_config": "wikitext2_train_joined"
        },
        "cs": { "type": "CaptureSplitInfo", "num_splits": 4, "unique_embeds_lm_head_splits": true },
        "mb": {
            "type": "ModelBuilder",
            "precision": "int4",
            "int4_block_size": 32,
            "int4_accuracy_level": 4,
            "int4_op_types_to_quantize": [ "MatMul", "Gather" ]
        },
        "mq": {
            "type": "MatMulNBitsToQDQ",
            "use_int4": true,
            "add_zero_point": true,
            "nodes_to_exclude": [ "/lm_head/MatMul_Q4" ],
            "save_as_external_data": true
        },
        "gs": {
            "type": "GraphSurgeries",
            "surgeries": [
                { "surgeon": "RemoveRopeMultiCache" },
                { "surgeon": "AttentionMaskToSequenceLengths" },
                { "surgeon": "SimplifiedLayerNormToL2Norm" }
            ],
            "save_as_external_data": true
        },
        "sq": {
            "type": "OnnxStaticQuantization",
            "data_config": "wikitext2_train_act",
            "activation_type": "uint16",
            "precision": "uint8",
            "calibration_providers": [ "CUDAExecutionProvider" ],
            "quant_preprocess": true,
            "op_types_to_exclude": [ "GatherBlockQuantized", "GroupQueryAttention", "MatMulNBits" ],
            "save_as_external_data": true,
            "extra_option": { "CalibStridedMinMax": 4 }
        },
        "sp": { "type": "SplitModel" },
        "st": { "type": "StaticLLM", "batch_size": 1, "context_length": 64 },
        "cb": {
            "type": "EPContextBinaryGenerator",
            "provider_options": {
                "htp_performance_mode": "burst",
                "htp_graph_finalization_optimization_mode": "3",
                "soc_model": "60"
            },
            "weight_sharing": true
        },
        "cp": { "type": "ComposeOnnxModels" }
    },
    "target": "qnn_system",
    "log_severity_level": 1,
    "output_dir": "models/qwen_2.5_7b_Instruct",
    "cache_dir": "cache",
    "no_artifacts": true
}

Olive logs

(.venv) PS F:\AI\gaotong\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN> olive run --config config.json
[2025-10-11 20:18:08,696] [INFO] [run.py:99:run_engine] Running workflow default_workflow
[2025-10-11 20:18:08,801] [INFO] [cache.py:138:__init__] Using cache directory: F:\AI\gaotong\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN\cache\default_workflow
[2025-10-11 20:18:10,205] [INFO] [accelerator_creator.py:79:_fill_accelerators] the accelerator device is not specified. Inferred device: npu.
[2025-10-11 20:18:10,205] [INFO] [accelerator_creator.py:204:create_accelerators] Running workflow on accelerator specs: npu-qnn
[2025-10-11 20:18:10,239] [INFO] [engine.py:217:run] Running Olive on accelerator: npu-qnn
[2025-10-11 20:18:10,239] [INFO] [engine.py:853:_create_system] Creating target system ...
[2025-10-11 20:18:10,240] [INFO] [engine.py:856:_create_system] Target system created in 0.000999 seconds
[2025-10-11 20:18:10,241] [INFO] [engine.py:859:_create_system] Creating host system ...
[2025-10-11 20:18:10,251] [INFO] [engine.py:862:_create_system] Host system created in 0.009694 seconds
[2025-10-11 20:18:10,998] [WARNING] [config_utils.py:347:validate_config] Keys {'extra_option'} are not part of OnnxStaticQuantizationConfig. Ignoring them.
[2025-10-11 20:18:11,034] [INFO] [engine.py:685:_run_pass] Running pass q:quarot
[2025-10-11 20:18:11,035] [INFO] [cache.py:235:load_run_from_model_id] Loading run d18e18b5 from cache.
[2025-10-11 20:18:11,035] [INFO] [cache.py:195:load_model] Loading model d18e18b5 from cache.
[2025-10-11 20:18:11,036] [INFO] [engine.py:722:_run_pass] Loaded model from cache: d18e18b5
[2025-10-11 20:18:11,036] [INFO] [engine.py:685:_run_pass] Running pass g:gptqmodel
[2025-10-11 20:18:11,059] [INFO] [cache.py:235:load_run_from_model_id] Loading run 8d8de7f5 from cache.
[2025-10-11 20:18:11,078] [INFO] [cache.py:195:load_model] Loading model 8d8de7f5 from cache.
[2025-10-11 20:18:11,086] [INFO] [engine.py:722:_run_pass] Loaded model from cache: 8d8de7f5
[2025-10-11 20:18:11,086] [INFO] [engine.py:685:_run_pass] Running pass cs:capturesplitinfo
[2025-10-11 20:18:11,088] [INFO] [cache.py:235:load_run_from_model_id] Loading run 0cd3e38e from cache.
[2025-10-11 20:18:11,095] [INFO] [cache.py:195:load_model] Loading model 0cd3e38e from cache.
[2025-10-11 20:18:11,114] [INFO] [engine.py:722:_run_pass] Loaded model from cache: 0cd3e38e
[2025-10-11 20:18:11,115] [INFO] [engine.py:685:_run_pass] Running pass mb:modelbuilder
[2025-10-11 20:18:11,116] [INFO] [cache.py:235:load_run_from_model_id] Loading run 26fab61a from cache.
[2025-10-11 20:18:11,133] [INFO] [cache.py:195:load_model] Loading model 26fab61a from cache.
[2025-10-11 20:18:11,156] [INFO] [engine.py:722:_run_pass] Loaded model from cache: 26fab61a
[2025-10-11 20:18:11,156] [INFO] [engine.py:685:_run_pass] Running pass mq:matmulnbitstoqdq
[2025-10-11 20:18:11,157] [INFO] [cache.py:235:load_run_from_model_id] Loading run 6fefae8a from cache.
[2025-10-11 20:18:11,165] [INFO] [cache.py:195:load_model] Loading model 6fefae8a from cache.
[2025-10-11 20:18:11,182] [INFO] [engine.py:722:_run_pass] Loaded model from cache: 6fefae8a
[2025-10-11 20:18:11,183] [INFO] [engine.py:685:_run_pass] Running pass gs:graphsurgeries
[2025-10-11 20:18:11,185] [INFO] [cache.py:235:load_run_from_model_id] Loading run b7e6105a from cache.
[2025-10-11 20:18:11,192] [INFO] [cache.py:195:load_model] Loading model b7e6105a from cache.
[2025-10-11 20:18:11,203] [INFO] [engine.py:722:_run_pass] Loaded model from cache: b7e6105a
[2025-10-11 20:18:11,204] [INFO] [engine.py:685:_run_pass] Running pass sq:onnxstaticquantization
[2025-10-11 20:18:11,422] [INFO] [quantization.py:402:_run_for_config] Preprocessing model for quantization
2025-10-11 20:19:31.4721216 [E:onnxruntime:, sequential_executor.cc:572 onnxruntime::ExecuteKernel] Non-zero status code returned while running Mul node. Name:'/model/layers.14/mlp/act_fn/Mul' Status Message: E:\_work\1\s\onnxruntime\core\framework\bfc_arena.cc:359 onnxruntime::BFCArena::AllocateRawInternal Failed to allocate memory for requested buffer of size 159383552

[2025-10-11 20:19:31,579] [ERROR] [engine.py:748:_run_pass] Pass run failed.
Traceback (most recent call last):
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 736, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\local.py", line 45, in run_pass
    output_model = the_pass.run(model, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\passes\olive_pass.py", line 242, in run
    output_model = self._run_for_config(model, self.config, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\passes\onnx\quantization.py", line 462, in _run_for_config
    quantize_static(
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\onnxruntime\quantization\quantize.py", line 742, in quantize_static
    calibrator.collect_data(calibration_data_reader)
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\onnxruntime\quantization\calibrate.py", line 420, in collect_data
    self.intermediate_outputs.append(self.infer_session.run(None, inputs))
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 275, in run
    return self._sess.run(output_names, input_feed, run_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Mul node. Name:'/model/layers.14/mlp/act_fn/Mul' Status Message: E:\_work\1\s\onnxruntime\core\framework\bfc_arena.cc:359 onnxruntime::BFCArena::AllocateRawInternal Failed to allocate memory for requested buffer of size 159383552

[2025-10-11 20:19:31,631] [WARNING] [engine.py:310:run_accelerator] Failed to run Olive on npu-qnn.
Traceback (most recent call last):
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 306, in run_accelerator
    output_footprint = self._run_no_search(input_model_config, input_model_id, accelerator_spec, output_dir)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 350, in _run_no_search
    should_prune, signal, model_ids = self._run_passes(input_model_config, input_model_id, accelerator_spec)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 641, in _run_passes
    model_config, model_id = self._run_pass(
                             ^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 736, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\local.py", line 45, in run_pass
    output_model = the_pass.run(model, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\passes\olive_pass.py", line 242, in run
    output_model = self._run_for_config(model, self.config, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\passes\onnx\quantization.py", line 462, in _run_for_config
    quantize_static(
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\onnxruntime\quantization\quantize.py", line 742, in quantize_static
    calibrator.collect_data(calibration_data_reader)
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\onnxruntime\quantization\calibrate.py", line 420, in collect_data
    self.intermediate_outputs.append(self.infer_session.run(None, inputs))
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 275, in run
    return self._sess.run(output_names, input_feed, run_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Mul node. Name:'/model/layers.14/mlp/act_fn/Mul' Status Message: E:\_work\1\s\onnxruntime\core\framework\bfc_arena.cc:359 onnxruntime::BFCArena::AllocateRawInternal Failed to allocate memory for requested buffer of size 159383552

[2025-10-11 20:19:36,148] [INFO] [engine.py:235:run] Run history for npu-qnn:
[2025-10-11 20:19:36,175] [INFO] [engine.py:499:dump_run_history] run history:
+------------+-------------------+------------------+----------------+-----------+
| model_id   | parent_model_id   | from_pass        |   duration_sec | metrics   |
+============+===================+==================+================+===========+
| d4993402   |                   |                  |                |           |
+------------+-------------------+------------------+----------------+-----------+
| d18e18b5   | d4993402          | quarot           |      0.0019691 |           |
+------------+-------------------+------------------+----------------+-----------+
| 8d8de7f5   | d18e18b5          | gptqmodel        |      0.0498719 |           |
+------------+-------------------+------------------+----------------+-----------+
| 0cd3e38e   | 8d8de7f5          | capturesplitinfo |      0.027631  |           |
+------------+-------------------+------------------+----------------+-----------+
| 26fab61a   | 0cd3e38e          | modelbuilder     |      0.0411398 |           |
+------------+-------------------+------------------+----------------+-----------+
| 6fefae8a   | 26fab61a          | matmulnbitstoqdq |      0.0255711 |           |
+------------+-------------------+------------------+----------------+-----------+
| b7e6105a   | 6fefae8a          | graphsurgeries   |      0.0201991 |           |
+------------+-------------------+------------------+----------------+-----------+
[2025-10-11 20:19:36,175] [WARNING] [engine.py:259:run] No output model
No output model produced. Please check the log for details.

Other information

• OS: Windows 11 Professional Edition 25H2 26220.6772 x64​

Package                Version      Editable project location
---------------------- ------------ -----------------------------
accelerate             1.10.1
aiohappyeyeballs       2.6.1
aiohttp                3.13.0
aiosignal              1.4.0
alembic                1.16.5
annotated-types        0.7.0
anyio                  4.11.0
attrs                  25.4.0
auto_gptq              0.8.0.dev0   F:\AI\gaotong\olive\AutoGPTQ
autopep8               2.3.2
certifi                2025.10.5
cffi                   2.0.0
charset-normalizer     3.4.3
colorama               0.4.6
coloredlogs            15.0.1
colorlog               6.9.0
datasets               4.2.0
device-smi             0.4.1
dill                   0.4.0
filelock               3.20.0
flatbuffers            25.9.23
frozenlist             1.8.0
fsspec                 2025.9.0
gekko                  1.3.0
gptqmodel              4.0.0.dev0   F:\AI\gaotong\olive\GPTQModel
greenlet               3.2.4
h11                    0.16.0
hf_transfer            0.1.9
httpcore               1.0.9
httpx                  0.28.1
huggingface-hub        0.35.3
humanfriendly          10.0
idna                   3.10
iniconfig              2.1.0
Jinja2                 3.1.6
lightning-utilities    0.15.2
logbar                 0.0.4
Mako                   1.3.10
MarkupSafe             3.0.3
ml_dtypes              0.5.3
mpmath                 1.3.0
multidict              6.7.0
multiprocess           0.70.16
networkx               3.5
numpy                  2.3.3
olive-ai               0.9.3
onnx                   1.19.1
onnx-ir                0.1.10
onnxruntime-genai-cuda 0.9.2
onnxruntime-gpu        1.23.0
onnxscript             0.5.3
optimum                2.0.0
optuna                 4.5.0
packaging              25.0
pandas                 2.3.3
peft                   0.17.1
pillow                 11.3.0
pip                    25.2
pluggy                 1.6.0
propcache              0.4.1
protobuf               6.32.1
psutil                 7.1.0
pyarrow                21.0.0
pycodestyle            2.14.0
pycparser              2.23
pydantic               2.12.0
pydantic_core          2.41.1
Pygments               2.19.2
pyreadline3            3.5.4
pytest                 8.4.2
python-dateutil        2.9.0.post0
pytz                   2025.2
PyYAML                 6.0.3
random_word            1.0.13
regex                  2025.9.18
requests               2.32.5
rouge                  1.0.1
safetensors            0.6.2
sentencepiece          0.2.1
setuptools             80.9.0
six                    1.17.0
sniffio                1.3.1
soundfile              0.13.1
SQLAlchemy             2.0.43
sympy                  1.13.1
tabulate               0.9.0
threadpoolctl          3.6.0
tokenicer              0.0.4
tokenizers             0.21.4
torch                  2.5.1+cu121
torchaudio             2.5.1+cu121
torchmetrics           1.8.2
torchvision            0.20.1+cu121
tqdm                   4.67.1
transformers           4.53.2
typing_extensions      4.15.0
typing-inspection      0.4.2
tzdata                 2025.2
urllib3                2.5.0
xxhash                 3.6.0
yarl                   1.22.0

QNN Environment Package List:​

Package             Version
------------------- -----------
alembic             1.16.5
annotated-types     0.7.0
certifi             2025.10.5
charset-normalizer  3.4.3
colorama            0.4.6
coloredlogs         15.0.1
colorlog            6.9.0
filelock            3.20.0
flatbuffers         25.9.23
fsspec              2025.9.0
greenlet            3.2.4
huggingface-hub     0.35.3
humanfriendly       10.0
idna                3.10
Jinja2              3.1.6
lightning-utilities 0.15.2
Mako                1.3.10
MarkupSafe          3.0.3
ml_dtypes           0.5.3
mpmath              1.3.0
networkx            3.5
numpy               2.3.3
olive-ai            0.9.3
onnx                1.19.1
onnx-ir             0.1.10
onnxruntime-qnn     1.22.2
onnxscript          0.5.3
optuna              4.5.0
packaging           25.0
pandas              2.3.3
pip                 25.2
protobuf            6.32.1
pydantic            2.12.0
pydantic_core       2.41.1
pyreadline3         3.5.4
python-dateutil     2.9.0.post0
pytz                2025.2
PyYAML              6.0.3
regex               2025.9.18
requests            2.32.5
safetensors         0.6.2
setuptools          80.9.0
six                 1.17.0
SQLAlchemy          2.0.43
sympy               1.14.0
tokenizers          0.22.1
torch               2.8.0
torchmetrics        1.8.2
tqdm                4.67.1
transformers        4.57.0
typing_extensions   4.15.0
typing-inspection   0.4.2
tzdata              2025.2
urllib3             2.5.0

Additional context
Computer Configuration 1:​
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (3.70 GHz)​
RAM: 32GB​
GPU: NVIDIA GeForce GTX 1080 Ti 11G​
CUDA: 12.1​

Computer Configuration 2:​
CPU: Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz (2.10 GHz)​
RAM: 256GB​
GPU: NVIDIA GeForce RTX 4090 24G​
CUDA: 12.8

[nnbw-liu]

Qwen2.5-7B-Instruct

config:

{
    "input_model": { "type": "HfModel", "model_path": "Qwen/Qwen2.5-7B-Instruct" },
    "systems": {
        "qnn_system": {
            "type": "PythonEnvironment",
            "python_environment_path": "D:\\olive\\qnn-env\\.venv\\Scripts",
            "accelerators": [ { "execution_providers": [ "QNNExecutionProvider" ] } ]
        }
    },
    "data_configs": [
        {
            "name": "wikitext2_train_joined",
            "type": "HuggingfaceContainer",
            "load_dataset_config": { "data_name": "wikitext", "subset": "wikitext-2-raw-v1", "split": "train" },
            "pre_process_data_config": {
                "strategy": "join",
                "add_special_tokens": false,
                "max_seq_len": 4096,
                "max_samples": 128
            }
        },
        {
            "name": "wikitext2_train_act",
            "type": "HuggingfaceContainer",
            "load_dataset_config": { "data_name": "wikitext", "subset": "wikitext-2-raw-v1", "split": "train" },
            "pre_process_data_config": {
                "strategy": "line-by-line",
                "add_special_tokens": true,
                "max_samples": 256,
                "max_seq_len": 4096
            }
        }
    ],
    "passes": {
        "q": { "type": "QuaRot" },
        "g": {
            "type": "GptqModel",
            "bits": 4,
            "sym": true,
            "group_size": -1,
            "lm_head": false,
            "device": "cuda",
            "data_config": "wikitext2_train_joined"
        },
        "cs": { "type": "CaptureSplitInfo", "num_splits": 4, "unique_embeds_lm_head_splits": true },
        "mb": {
            "type": "ModelBuilder",
            "precision": "int4",
            "int4_block_size": 32,
            "int4_accuracy_level": 4,
            "int4_op_types_to_quantize": [ "MatMul", "Gather" ]
        },
        "mq": {
            "type": "MatMulNBitsToQDQ",
            "use_int4": true,
            "add_zero_point": true,
            "nodes_to_exclude": [ "/lm_head/MatMul_Q4" ],
            "save_as_external_data": true
        },
        "gs": {
            "type": "GraphSurgeries",
            "surgeries": [
                { "surgeon": "RemoveRopeMultiCache" },
                { "surgeon": "AttentionMaskToSequenceLengths" },
                { "surgeon": "SimplifiedLayerNormToL2Norm" }
            ],
            "save_as_external_data": true
        },
        "sq": {
            "type": "OnnxStaticQuantization",
            "data_config": "wikitext2_train_act",
            "activation_type": "uint16",
            "precision": "uint8",
            "calibration_providers": [ "CUDAExecutionProvider" ],
            "quant_preprocess": true,
            "op_types_to_exclude": [ "GatherBlockQuantized", "GroupQueryAttention", "MatMulNBits" ],
            "save_as_external_data": true,
            "extra_option": { "CalibStridedMinMax": 4 }
        },
        "sp": { "type": "SplitModel" },
        "st": { "type": "StaticLLM", "batch_size": 1, "context_length": 64 },
        "cb": {
            "type": "EPContextBinaryGenerator",
            "provider_options": {
                "htp_performance_mode": "burst",
                "htp_graph_finalization_optimization_mode": "3",
                "soc_model": "60"
            },
            "weight_sharing": true
        },
        "cp": { "type": "ComposeOnnxModels" }
    },
    "target": "qnn_system",
    "log_severity_level": 1,
    "output_dir": "models/qwen_2.5_7b_Instruct",
    "cache_dir": "cache",
    "no_artifacts": true
}

log:

(.venv) PS D:\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN> olive run --config config.json
[2025-10-11 15:09:44,080] [INFO] [run.py:99:run_engine] Running workflow default_workflow
[2025-10-11 15:09:44,155] [INFO] [cache.py:138:__init__] Using cache directory: D:\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN\cache\default_workflow
[2025-10-11 15:09:45,116] [INFO] [accelerator_creator.py:79:_fill_accelerators] the accelerator device is not specified. Inferred device: npu.
[2025-10-11 15:09:45,118] [INFO] [accelerator_creator.py:204:create_accelerators] Running workflow on accelerator specs: npu-qnn
[2025-10-11 15:09:45,168] [INFO] [engine.py:217:run] Running Olive on accelerator: npu-qnn
[2025-10-11 15:09:45,169] [INFO] [engine.py:853:_create_system] Creating target system ...
[2025-10-11 15:09:45,173] [INFO] [engine.py:856:_create_system] Target system created in 0.003817 seconds
[2025-10-11 15:09:45,174] [INFO] [engine.py:859:_create_system] Creating host system ...
[2025-10-11 15:09:45,179] [INFO] [engine.py:862:_create_system] Host system created in 0.002456 seconds
[2025-10-11 15:09:46,420] [INFO] [engine.py:685:_run_pass] Running pass q:quarot
[2025-10-11 15:09:46,423] [INFO] [cache.py:235:load_run_from_model_id] Loading run 60dcbba8 from cache.
[2025-10-11 15:09:46,425] [INFO] [cache.py:195:load_model] Loading model 60dcbba8 from cache.
[2025-10-11 15:09:46,428] [INFO] [engine.py:722:_run_pass] Loaded model from cache: 60dcbba8
[2025-10-11 15:09:46,428] [INFO] [engine.py:685:_run_pass] Running pass g:gptqmodel
[2025-10-11 15:09:46,431] [INFO] [cache.py:235:load_run_from_model_id] Loading run b6d84b17 from cache.
[2025-10-11 15:09:46,433] [INFO] [cache.py:195:load_model] Loading model b6d84b17 from cache.
[2025-10-11 15:09:46,436] [INFO] [engine.py:722:_run_pass] Loaded model from cache: b6d84b17
[2025-10-11 15:09:46,437] [INFO] [engine.py:685:_run_pass] Running pass cs:capturesplitinfo
[2025-10-11 15:09:46,439] [INFO] [cache.py:235:load_run_from_model_id] Loading run edb7dfde from cache.
[2025-10-11 15:09:46,441] [INFO] [cache.py:195:load_model] Loading model edb7dfde from cache.
[2025-10-11 15:09:46,444] [INFO] [engine.py:722:_run_pass] Loaded model from cache: edb7dfde
[2025-10-11 15:09:46,445] [INFO] [engine.py:685:_run_pass] Running pass mb:modelbuilder
[2025-10-11 15:09:46,448] [INFO] [cache.py:235:load_run_from_model_id] Loading run 24ac0135 from cache.
[2025-10-11 15:09:46,450] [INFO] [cache.py:195:load_model] Loading model 24ac0135 from cache.
[2025-10-11 15:09:46,454] [INFO] [engine.py:722:_run_pass] Loaded model from cache: 24ac0135
[2025-10-11 15:09:46,457] [INFO] [engine.py:685:_run_pass] Running pass mq:matmulnbitstoqdq
[2025-10-11 15:09:46,459] [INFO] [cache.py:235:load_run_from_model_id] Loading run ec61f762 from cache.
[2025-10-11 15:09:46,461] [INFO] [cache.py:195:load_model] Loading model ec61f762 from cache.
[2025-10-11 15:09:46,466] [INFO] [engine.py:722:_run_pass] Loaded model from cache: ec61f762
[2025-10-11 15:09:46,469] [INFO] [engine.py:685:_run_pass] Running pass gs:graphsurgeries
[2025-10-11 15:09:46,471] [INFO] [cache.py:235:load_run_from_model_id] Loading run 8d19c5e9 from cache.
[2025-10-11 15:09:46,473] [INFO] [cache.py:195:load_model] Loading model 8d19c5e9 from cache.
[2025-10-11 15:09:46,477] [INFO] [engine.py:722:_run_pass] Loaded model from cache: 8d19c5e9
[2025-10-11 15:09:46,479] [INFO] [engine.py:685:_run_pass] Running pass sq:onnxstaticquantization
[2025-10-11 15:09:46,913] [INFO] [quantization.py:402:_run_for_config] Preprocessing model for quantization
2025-10-11 15:12:38.5183683 [E:onnxruntime:, sequential_executor.cc:572 onnxruntime::ExecuteKernel] Non-zero status code returned while running DequantizeLinear node. Name:'/model/layers.0/attn/qkv_proj/DequantizeLinear' Status Message: CUDA error cudaErrorUnknown:unknown error
Unhandled exception caught in c10/util/AbortHandler.h
00007FFDADB3E8B400007FFDADB36050 torch_python.dll!torch::autograd::THPCppFunction_requires_grad [<unknown file> @ <unknown line number>]
00007FFE865119D700007FFE865119C0 ucrtbase.dll!terminate [<unknown file> @ <unknown line number>]
00007FFE7F1C1911 <unknown symbol address> VCRUNTIME140_1.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFE7F1C218F <unknown symbol address> VCRUNTIME140_1.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFE7F1C21E9 <unknown symbol address> VCRUNTIME140_1.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFE7F1C401900007FFE7F1C3F70 VCRUNTIME140_1.dll!_CxxFrameHandler4 [<unknown file> @ <unknown line number>]
00007FFE8942796F00007FFE894278D0 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FFE8933639700007FFE89335780 ntdll.dll!RtlWow64GetCurrentCpuArea [<unknown file> @ <unknown line number>]
00007FFE8933522600007FFE89335020 ntdll.dll!RtlRaiseException [<unknown file> @ <unknown line number>]
00007FFE869C66CA00007FFE869C6640 KERNELBASE.dll!RaiseException [<unknown file> @ <unknown line number>]
00007FFE8156526700007FFE815651D0 VCRUNTIME140.dll!CxxThrowException [<unknown file> @ <unknown line number>]
00007FFCDB39D6B900007FFCDB097710 onnxruntime_providers_cuda.dll!GetProvider [<unknown file> @ <unknown line number>]
00007FFE7F1C416000007FFE7F1C3F70 VCRUNTIME140_1.dll!_CxxFrameHandler4 [<unknown file> @ <unknown line number>]
00007FFE7F1C2572 <unknown symbol address> VCRUNTIME140_1.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFE893E2CC600007FFE893E2820 ntdll.dll!RtlCaptureContext2 [<unknown file> @ <unknown line number>]
00007FFCDB07B8A5 <unknown symbol address> onnxruntime_providers_cuda.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFCDB097F4F00007FFCDB097710 onnxruntime_providers_cuda.dll!GetProvider [<unknown file> @ <unknown line number>]
00007FFD76A9EE8900007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD76A972E900007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD76A17C0A00007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD7637773D00007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD76379A1100007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD7630FB6600007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD76336F9100007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD7633703400007FFD762C51A0 onnxruntime_pybind11_state.pyd!PyInit_onnxruntime_pybind11_state [<unknown file> @ <unknown line number>]
00007FFD7629CAD2 <unknown symbol address> onnxruntime_pybind11_state.pyd!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFDE93E72F800007FFDE93E6C24 python312.dll!PyUnicode_DecodeUTF7Stateful [<unknown file> @ <unknown line number>]
00007FFDE9416A0200007FFDE9416380 python312.dll!PyObject_Vectorcall [<unknown file> @ <unknown line number>]
00007FFDE94163B500007FFDE9416380 python312.dll!PyObject_Vectorcall [<unknown file> @ <unknown line number>]
00007FFDE94178AD00007FFDE9416FE0 python312.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFDE942A74C00007FFDE942A020 python312.dll!PyObject_ClearManagedDict [<unknown file> @ <unknown line number>]
00007FFDE9429F7D00007FFDE9429ED0 python312.dll!PyEval_EvalCode [<unknown file> @ <unknown line number>]
00007FFDE93E87A800007FFDE93E837C python312.dll!PyWeakref_NewProxy [<unknown file> @ <unknown line number>]
00007FFDE93E864C00007FFDE93E837C python312.dll!PyWeakref_NewProxy [<unknown file> @ <unknown line number>]
00007FFDE93F984E00007FFDE93F9240 python312.dll!PySet_Add [<unknown file> @ <unknown line number>]
00007FFDE941644F00007FFDE9416380 python312.dll!PyObject_Vectorcall [<unknown file> @ <unknown line number>]
00007FFDE94163B500007FFDE9416380 python312.dll!PyObject_Vectorcall [<unknown file> @ <unknown line number>]
00007FFDE94178AD00007FFDE9416FE0 python312.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFDE9415A2C00007FFDE94158B0 python312.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFDE948EC7500007FFDE948EB50 python312.dll!PyObject_Call [<unknown file> @ <unknown line number>]
00007FFDE948EBBF00007FFDE948EB50 python312.dll!PyObject_Call [<unknown file> @ <unknown line number>]
00007FFDE94D6A2000007FFDE94D68F4 python312.dll!Py_ExitStatusException [<unknown file> @ <unknown line number>]
00007FFDE94D5DA500007FFDE94D5BA0 python312.dll!Py_RunMain [<unknown file> @ <unknown line number>]
00007FFDE94D5BB500007FFDE94D5BA0 python312.dll!Py_RunMain [<unknown file> @ <unknown line number>]
00007FFDE94D62B100007FFDE94D628C python312.dll!Py_Main [<unknown file> @ <unknown line number>]
00007FF626F21230 <unknown symbol address> python.exe!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFE87DAE8D700007FFE87DAE8C0 KERNEL32.DLL!BaseThreadInitThunk [<unknown file> @ <unknown line number>]
00007FFE892C8D9C00007FFE892C8D70 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]

[xieofxie]

it is usually because your GPU has not enough memory. You could remove "calibration_providers": [ "CUDAExecutionProvider" ], to make it totally on CPU (which is slow) and try again

[nnbw-liu]

it is usually because your GPU has not enough memory. You could remove to make it totally on CPU (which is slow) and try again"calibration_providers": [ "CUDAExecutionProvider" ],

@xieofxie Thank you, I'll give it a try first. But there is a question: Previously, CUDA could be used for successful conversion on VS code, but now it is not possible, even if the max_seq_len and max_samples in the dataset are reduced

[nnbw-liu]

@xieofxie
after remove "calibration_providers": [ "CUDAExecutionProvider" ],

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MatMulNBits node. Name:'/lm_head/MatMul_Q4' Status Message: bad allocation

Can you help me adjust the configuration file according to my computer configuration?

[xieofxie]

bad allocation. Maybe you need to increase your virtual memory.

I don't know what your device is.. Usually 64GB mem with 32GB shared memory GPU should work (this is how we tested it)

I mean models like microsoft_Phi-3.5-mini-instruct, deepseek-ai_DeepSeek-R1-Distill-Qwen-1.5B, Qwen_Qwen2.5-1.5B-Instruct. But since your are converting 4B or 7B, they surely need more memroy

[nnbw-liu]

bad allocation. Maybe you need to increase your virtual memory.

I don't know what your device is.. Usually 64GB mem with 32GB shared memory GPU should work (this is how we tested it)

I mean models like microsoft_Phi-3.5-mini-instruct, deepseek-ai_DeepSeek-R1-Distill-Qwen-1.5B, Qwen_Qwen2.5-1.5B-Instruct. But since your are converting 4B or 7B, they surely need more memroy

@xieofxie
Computer Configuration 1:​
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (3.70 GHz)​
RAM: 32GB​
GPU: NVIDIA GeForce GTX 1080 Ti 11G​
CUDA: 12.1​

Computer Configuration 2:​
CPU: Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz (2.10 GHz)​
RAM: 256GB​
GPU: NVIDIA GeForce RTX 4090 24G​
CUDA: 12.8

[nnbw-liu]

@xieofxie

{
            "name": "wikitext2_train_act",
            "type": "HuggingfaceContainer",
            "load_dataset_config": { "data_name": "wikitext", "subset": "wikitext-2-raw-v1", "split": "train" },
            "pre_process_data_config": {
                "strategy": "line-by-line",
                "add_special_tokens": true,
                "max_samples": 128,
                "max_seq_len": 512
            }
        }

After lowering the parameters, the following error occurred

[2025-10-13 15:04:29,145] [INFO] [engine.py:685:_run_pass] Running pass cb:epcontextbinarygenerator
[2025-10-13 15:05:19,027] [ERROR] [utils.py:73:run_subprocess] Failed to run ['F:\\AI\\gaotong\\olive\\qnn\\.venv\\Scripts\\python.EXE', 'F:\\AI\\gaotong\\olive\\.venv\\Lib\\site-packages\\olive\\systems\\python_environment\\pass_runner.py', '--model_config', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\model_config.json', '--pass_config', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\pass_config.json', '--tempdir', 'C:\\Users\\12105\\AppData\\Local\\Temp', '--output_model_path', 'F:\\AI\\gaotong\\olive\\olive-recipes\\Qwen-Qwen2.5-7B-Instruct\\QNN\\cache\\default_workflow\\runs\\f95f72e5\\models', '--output_path', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\output.json'] with returncode 1!
Stderr: Traceback (most recent call last):
  File "F:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\python_environment\pass_runner.py", line 53, in <module>
    main()
  File "F:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\python_environment\pass_runner.py", line 45, in main
    output_model = the_pass.run(model, args.output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\olive\passes\olive_pass.py", line 242, in run
    output_model = self._run_for_config(model, self.config, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\olive\passes\onnx\context_binary.py", line 150, in _run_for_config
    return process_llm_pipeline(
           ^^^^^^^^^^^^^^^^^^^^^
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\olive\passes\onnx\common.py", line 568, in process_llm_pipeline
    new_groups = process_func(component_models, llm_pipeline, output_dir)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\olive\passes\onnx\context_binary.py", line 125, in process_context_iterator
    composite_binaries = self._generate_composite_binaries(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\olive\passes\onnx\context_binary.py", line 198, in _generate_composite_binaries
    new_models[model_name] = cls._generate_context_binary(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\olive\passes\onnx\context_binary.py", line 275, in _generate_context_binary
    ort.InferenceSession(
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 472, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "F:\AI\gaotong\olive\qnn\.venv\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 550, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from F:\AI\gaotong\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN\cache\default_workflow\runs\b7fd2f9f\models\context_0.onnx failed:This is an invalid model. In Node, ("/model/layers.0/attn/GroupQueryAttention", GroupQueryAttention, "com.microsoft", -1) : ("/model/layers.0/attn/q_norm/Reshape_2/output_0_DequantizeLinear_Output": tensor(float),"/model/layers.0/attn/k_norm/Reshape_2/output_0_DequantizeLinear_Output": tensor(float),"/model/layers.0/attn/v_proj/MatMul/output_0_DequantizeLinear_Output": tensor(float),"past_key_values.0.key": tensor(float),"past_key_values.0.value": tensor(float),"past_seq_len": tensor(int32),"total_seq_len": tensor(int32),"cos_cache": tensor(float),"sin_cache": tensor(float),"","",) -> ("/model/layers.0/attn/GroupQueryAttention/output_0": tensor(float),"present.0.key": tensor(float),"present.0.value": tensor(float),) , Error Unrecognized attribute: qk_output for operator GroupQueryAttention

Stdout:
Env: environ({'ACSVCPORT': '17532', 'ALLUSERSPROFILE': 'C:\\ProgramData', 'APPDATA': 'C:\\Users\\12105\\AppData\\Roaming', 'AQUA_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\aqua.vmoptions', 'CHOCOLATEYINSTALL': 'C:\\ProgramData\\chocolatey', 'CHOCOLATEYLASTPATHUPDATE': '133645441563375655', 'CHROME_CRASHPAD_PIPE_NAME': '\\\\.\\pipe\\crashpad_1116_KFFRLSCLHWNSUUWX', 'CLION_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\clion.vmoptions', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files', 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', 'COMPUTERNAME': 'PC-8700K', 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', 'CUDA_PATH': 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1', 'CUDA_PATH_V12_1': 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1', 'DATAGRIP_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\datagrip.vmoptions', 'DATASPELL_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\dataspell.vmoptions', 'DEVECO STUDIO': 'C:\\Program Files\\Huawei\\DevEco Studio\\bin;', 'DEVECOSTUDIO_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\devecostudio.vmoptions', 'DRIVERDATA': 'C:\\Windows\\System32\\Drivers\\DriverData', 'EFC_9468_1262719628': '1', 'EFC_9468_1592913036': '1', 'EFC_9468_2283032206': '1', 'EFC_9468_2775293581': '1', 'EFC_9468_3789132940': '1', 'FPS_BROWSER_APP_PROFILE_STRING': 'Internet Explorer', 'FPS_BROWSER_USER_PROFILE_STRING': 'Default', 'GATEWAY_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\gateway.vmoptions', 'GIT_LFS_PATH': 'C:\\Program Files\\Git LFS', 'GOLAND_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\goland.vmoptions', 'HOMEDRIVE': 'C:', 'HOMEPATH': '\\Users\\12105', 'IDEA_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\idea.vmoptions', 'INFERENCEENGINE_DIR': 'F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\cmake', 'INTELLIJ IDEA': 'D:\\Program Files\\JetBrains\\IntelliJ IDEA 2024.2.4\\bin;', 'INTELLIJ IDEA COMMUNITY EDITION': 'C:\\Program Files\\JetBrains\\IntelliJ IDEA Community Edition 2024.2.4\\bin;', 'INTEL_OPENVINO_DIR': 'F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0', 'JETBRAINSCLIENT_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\jetbrainsclient.vmoptions', 'JETBRAINS_CLIENT_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\jetbrains_client.vmoptions', 'KMP_DUPLICATE_LIB_OK': 'TRUE', 'LOCALAPPDATA': 'C:\\Users\\12105\\AppData\\Local', 'LOGONSERVER': '\\\\PC-8700K', 'NGRAPH_DIR': 'F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\cmake', 'NUMBER_OF_PROCESSORS': '12', 'ONEDRIVE': 'C:\\Users\\12105\\OneDrive', 'ONEDRIVECONSUMER': 'C:\\Users\\12105\\OneDrive', 'OPENVINO_DIR': 'F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\cmake', 'OPENVINO_LIB_PATHS': 'F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\bin;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\bin\\intel64\\vc14;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\redist\\intel64\\vc14;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\bin\\intel64\\Release;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\bin\\intel64\\Debug;', 'ORIGINAL_XDG_CURRENT_DESKTOP': 'undefined', 'OS': 'Windows_NT', 'PATH': 'F:\\AI\\gaotong\\olive\\qnn\\.venv\\Scripts;F:\\AI\\gaotong\\olive\\.venv\\Scripts;C:\\Users\\12105\\.dotnet\\tools\\.store\\powershell\\7.4.4\\powershell\\7.4.4\\tools\\net8.0\\any\\win;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1\\libnvvp;;C:\\Program Files (x86)\\Razer Chroma SDK\\bin;C:\\Program Files\\Razer Chroma SDK\\bin;C:\\Program Files (x86)\\Razer\\ChromaBroadcast\\bin;C:\\Program Files\\Razer\\ChromaBroadcast\\bin;C:\\Program Files\\Microsoft\\jdk-11.0.12.7-hotspot\\bin;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Program Files\\Microsoft SQL Server\\Client SDK\\ODBC\\170\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Users\\12105\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\12105\\.dotnet\\tools;C:\\Program Files\\Git\\cmd;C:\\Windows\\system32\\config\\systemprofile\\AppData\\Local\\Microsoft\\WindowsApps;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;F:\\mysql-8.0.11-winx64\\bin;F:\\Huawei\\ohpm\\bin;C:\\Program Files\\Redis\\;C:\\Program Files\\nodejs\\;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\CMake\\bin;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\bin;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\bin\\intel64\\vc14;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\redist\\intel64\\vc14;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\bin\\intel64\\Release;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\bin\\intel64\\Debug;C:\\nuget;C:\\Program Files\\Git LFS;C:\\Program Files\\NVIDIA Corporation\\NVIDIA NvDLISR;C:\\Program Files\\dotnet\\;C:\\Program Files\\NVIDIA Corporation\\NVIDIA app\\NvDLISR;C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2023.1.1\\;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python311\\Scripts\\;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python311\\;C:\\Users\\12105\\.cargo\\bin;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python312\\Scripts\\;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python312\\;C:\\Users\\12105\\.local\\bin;C:\\Users\\12105\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Program Files\\Huawei\\DevEco Studio\\bin;;C:\\Users\\12105\\AppData\\Local\\GitHubDesktop\\bin;C:\\Program Files\\JetBrains\\WebStorm 2024.1.5\\bin;;C:\\Users\\12105\\AppData\\Local\\Microsoft\\WinGet\\Links;C:\\Users\\12105\\AppData\\Roaming\\npm;D:\\Program Files\\JetBrains\\IntelliJ IDEA 2024.2.4\\bin;;C:\\Program Files\\JetBrains\\IntelliJ IDEA Community Edition 2024.2.4\\bin;;C:\\Users\\12105\\.dotnet\\tools;;c:\\Users\\12105\\AppData\\Roaming\\Code\\User\\globalStorage\\github.copilot-chat\\debugCommand', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL', 'PHPSTORM_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\phpstorm.vmoptions', 'PROCESSOR_ARCHITECTURE': 'AMD64', 'PROCESSOR_IDENTIFIER': 'Intel64 Family 6 Model 158 Stepping 10, GenuineIntel', 'PROCESSOR_LEVEL': '6', 'PROCESSOR_REVISION': '9e0a', 'PROGRAMDATA': 'C:\\ProgramData', 'PROGRAMFILES': 'C:\\Program Files', 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)', 'PROGRAMW6432': 'C:\\Program Files', 'PSMODULEPATH': 'D:\\Documents\\PowerShell\\Modules;C:\\Program Files\\PowerShell\\Modules;c:\\users\\12105\\.dotnet\\tools\\.store\\powershell\\7.4.4\\powershell\\7.4.4\\tools\\net8.0\\any\\win\\Modules;C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules', 'PUBLIC': 'C:\\Users\\Public', 'PYCHARM_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\pycharm.vmoptions', 'PYTHONPATH': 'F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\python;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\python\\python3;', 'QNN_SDK_ROOT': 'D:\\Downloads\\v2.38.0.250901\\qairt\\2.38.0.250901', 'RIDER_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\rider.vmoptions', 'RLSSVCPORT': '22112', 'RUBYMINE_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\rubymine.vmoptions', 'RUSTROVER_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\rustrover.vmoptions', 'SCRIPT_NAME': 'setupvars.bat', 'SESSIONNAME': 'Console', 'STUDIO_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\studio.vmoptions', 'SYSTEMDRIVE': 'C:', 'SYSTEMROOT': 'C:\\WINDOWS', 'TBB_DIR': 'F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\lib\\cmake\\tbb', 'TEMP': 'C:\\Users\\12105\\AppData\\Local\\Temp', 'TMP': 'C:\\Users\\12105\\AppData\\Local\\Temp', 'USERDOMAIN': 'PC-8700K', 'USERDOMAIN_ROAMINGPROFILE': 'PC-8700K', 'USERNAME': '12105', 'USERPROFILE': 'C:\\Users\\12105', 'VBOX_MSI_INSTALL_PATH': 'C:\\Program Files\\Oracle\\VirtualBox\\', 'VIRTUAL_ENV': 'F:\\AI\\gaotong\\olive\\.venv', 'VIRTUAL_ENV_PROMPT': '.venv', 'WEBIDE_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\webide.vmoptions', 'WEBSTORM': 'C:\\Program Files\\JetBrains\\WebStorm 2024.1.5\\bin;', 'WEBSTORM_VM_OPTIONS': 'D:\\Downloads\\jetbra-8f6785eac5e6e7e8b20e6174dd28bb19d8da7550\\jetbra\\vmoptions\\webstorm.vmoptions', 'WINDIR': 'C:\\WINDOWS', 'TERM_PROGRAM': 'vscode', 'TERM_PROGRAM_VERSION': '1.102.0', 'LANG': 'zh_CN.UTF-8', 'COLORTERM': 'truecolor', 'GIT_ASKPASS': 'c:\\Program Files\\Microsoft VS Code\\resources\\app\\extensions\\git\\dist\\askpass.sh', 'VSCODE_GIT_ASKPASS_NODE': 'C:\\Program Files\\Microsoft VS Code\\Code.exe', 'VSCODE_GIT_ASKPASS_EXTRA_ARGS': '', 'VSCODE_GIT_ASKPASS_MAIN': 'c:\\Program Files\\Microsoft VS Code\\resources\\app\\extensions\\git\\dist\\askpass-main.js', 'VSCODE_GIT_IPC_HANDLE': '\\\\.\\pipe\\vscode-git-9abd0d57c8-sock', 'PYTHONSTARTUP': 'c:\\Users\\12105\\AppData\\Roaming\\Code\\User\\workspaceStorage\\69b19362c2a60354c22d08c63aae5bf7\\ms-python.python\\pythonrc.py', 'PYTHON_BASIC_REPL': '1', 'VSCODE_INJECTION': '1', '_OLD_VIRTUAL_PATH': 'C:\\Users\\12105\\.dotnet\\tools\\.store\\powershell\\7.4.4\\powershell\\7.4.4\\tools\\net8.0\\any\\win;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1\\libnvvp;;C:\\Program Files (x86)\\Razer Chroma SDK\\bin;C:\\Program Files\\Razer Chroma SDK\\bin;C:\\Program Files (x86)\\Razer\\ChromaBroadcast\\bin;C:\\Program Files\\Razer\\ChromaBroadcast\\bin;C:\\Program Files\\Microsoft\\jdk-11.0.12.7-hotspot\\bin;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Program Files\\Microsoft SQL Server\\Client SDK\\ODBC\\170\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Users\\12105\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\12105\\.dotnet\\tools;C:\\Program Files\\Git\\cmd;C:\\Windows\\system32\\config\\systemprofile\\AppData\\Local\\Microsoft\\WindowsApps;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;F:\\mysql-8.0.11-winx64\\bin;F:\\Huawei\\ohpm\\bin;C:\\Program Files\\Redis\\;C:\\Program Files\\nodejs\\;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\CMake\\bin;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\bin;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\bin\\intel64\\vc14;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\3rdparty\\tbb\\redist\\intel64\\vc14;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\bin\\intel64\\Release;F:\\AI\\OpenVINO\\pkg\\openvino_genai_2024.6.0.0\\runtime\\bin\\intel64\\Debug;C:\\nuget;C:\\Program Files\\Git LFS;C:\\Program Files\\NVIDIA Corporation\\NVIDIA NvDLISR;C:\\Program Files\\dotnet\\;C:\\Program Files\\NVIDIA Corporation\\NVIDIA app\\NvDLISR;C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2023.1.1\\;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python311\\Scripts\\;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python311\\;C:\\Users\\12105\\.cargo\\bin;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python312\\Scripts\\;C:\\Users\\12105\\AppData\\Local\\Programs\\Python\\Python312\\;C:\\Users\\12105\\.local\\bin;C:\\Users\\12105\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Program Files\\Huawei\\DevEco Studio\\bin;;C:\\Users\\12105\\AppData\\Local\\GitHubDesktop\\bin;C:\\Program Files\\JetBrains\\WebStorm 2024.1.5\\bin;;C:\\Users\\12105\\AppData\\Local\\Microsoft\\WinGet\\Links;C:\\Users\\12105\\AppData\\Roaming\\npm;D:\\Program Files\\JetBrains\\IntelliJ IDEA 2024.2.4\\bin;;C:\\Program Files\\JetBrains\\IntelliJ IDEA Community Edition 2024.2.4\\bin;;C:\\Users\\12105\\.dotnet\\tools;;c:\\Users\\12105\\AppData\\Roaming\\Code\\User\\globalStorage\\github.copilot-chat\\debugCommand', 'OLIVE_CACHE_DIR': 'F:\\AI\\gaotong\\olive\\olive-recipes\\Qwen-Qwen2.5-7B-Instruct\\QNN\\cache\\default_workflow', 'OLIVE_LOG_LEVEL': 'INFO'})
[2025-10-13 15:05:19,031] [ERROR] [engine.py:748:_run_pass] Pass run failed.
Traceback (most recent call last):
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 736, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\python_environment\python_environment_system.py", line 107, in run_pass
    output_model_json = self._run_command(
                        ^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\python_environment\python_environment_system.py", line 87, in _run_command
    _, stdout, _ = run_subprocess(command, env=self.environ, check=True)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\common\utils.py", line 65, in run_subprocess
    out = subprocess.run(cmd, env=env, cwd=cwd, capture_output=True, check=check)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\12105\AppData\Local\Programs\Python\Python312\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['F:\\AI\\gaotong\\olive\\qnn\\.venv\\Scripts\\python.EXE', 'F:\\AI\\gaotong\\olive\\.venv\\Lib\\site-packages\\olive\\systems\\python_environment\\pass_runner.py', '--model_config', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\model_config.json', '--pass_config', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\pass_config.json', '--tempdir', 'C:\\Users\\12105\\AppData\\Local\\Temp', '--output_model_path', 'F:\\AI\\gaotong\\olive\\olive-recipes\\Qwen-Qwen2.5-7B-Instruct\\QNN\\cache\\default_workflow\\runs\\f95f72e5\\models', '--output_path', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\output.json']' returned non-zero exit status 1.
[2025-10-13 15:05:19,123] [WARNING] [engine.py:310:run_accelerator] Failed to run Olive on npu-qnn.
Traceback (most recent call last):
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 306, in run_accelerator
    output_footprint = self._run_no_search(input_model_config, input_model_id, accelerator_spec, output_dir)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 350, in _run_no_search
    should_prune, signal, model_ids = self._run_passes(input_model_config, input_model_id, accelerator_spec)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 641, in _run_passes
    model_config, model_id = self._run_pass(
                             ^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 736, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\python_environment\python_environment_system.py", line 107, in run_pass
    output_model_json = self._run_command(
                        ^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\systems\python_environment\python_environment_system.py", line 87, in _run_command
    _, stdout, _ = run_subprocess(command, env=self.environ, check=True)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\AI\gaotong\olive\.venv\Lib\site-packages\olive\common\utils.py", line 65, in run_subprocess
    out = subprocess.run(cmd, env=env, cwd=cwd, capture_output=True, check=check)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\12105\AppData\Local\Programs\Python\Python312\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['F:\\AI\\gaotong\\olive\\qnn\\.venv\\Scripts\\python.EXE', 'F:\\AI\\gaotong\\olive\\.venv\\Lib\\site-packages\\olive\\systems\\python_environment\\pass_runner.py', '--model_config', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\model_config.json', '--pass_config', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\pass_config.json', '--tempdir', 'C:\\Users\\12105\\AppData\\Local\\Temp', '--output_model_path', 'F:\\AI\\gaotong\\olive\\', 'F:\\AI\\gaotong\\olive\\olive-recipes\\Qwen-Qwen2.5-7B-Instruct\\QNN\\cache\\default_workflow\\runs\\f95f72e5\\models', '--output_path', 'C:\\Users\\12105\\AppData\\Local\\Temp\\tmpuqhwav6z\\output.json']' returned non-zero exit status 1.   
[2025-10-13 15:05:19,125] [INFO] [engine.py:235:run] Run history for npu-qnn:
[2025-10-13 15:05:19,153] [INFO] [engine.py:499:dump_run_history] run history:
+------------+-------------------+------------------------+----------------+-----------+
| model_id   | parent_model_id   | from_pass              |   duration_sec | metrics   |
+============+===================+========================+================+===========+
| d4993402   |                   |                        |                |           |
+------------+-------------------+------------------------+----------------+-----------+
| d18e18b5   | d4993402          | quarot                 |      0.0111642 |           |
+------------+-------------------+------------------------+----------------+-----------+
| 8d8de7f5   | d18e18b5          | gptqmodel              |      0.0220048 |           |
+------------+-------------------+------------------------+----------------+-----------+
| 0cd3e38e   | 8d8de7f5          | capturesplitinfo       |      0.014065  |           |
+------------+-------------------+------------------------+----------------+-----------+
| 26fab61a   | 0cd3e38e          | modelbuilder           |      0.026406  |           |
+------------+-------------------+------------------------+----------------+-----------+
| 6fefae8a   | 26fab61a          | matmulnbitstoqdq       |      0.0100739 |           |
+------------+-------------------+------------------------+----------------+-----------+
| b7e6105a   | 6fefae8a          | graphsurgeries         |      0.0342751 |           |
+------------+-------------------+------------------------+----------------+-----------+
| 9fd7133a   | b7e6105a          | onnxstaticquantization |   5965.98      |           |
+------------+-------------------+------------------------+----------------+-----------+
| 0b1010fc   | 9fd7133a          | splitmodel             |     26.5878    |           |
+------------+-------------------+------------------------+----------------+-----------+
| b7fd2f9f   | 0b1010fc          | staticllm              |     24.2291    |           |
+------------+-------------------+------------------------+----------------+-----------+
[2025-10-13 15:05:19,153] [WARNING] [engine.py:259:run] No output model
No output model produced. Please check the log for details.

[xieofxie]

it looks like the converted model is malform onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from F:\AI\gaotong\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN\cache\default_workflow\runs\b7fd2f9f\models\context_0.onnx failed:This is an invalid model. In Node, ("/model/layers.0/attn/GroupQueryAttention", GroupQueryAttention, "com.microsoft", -1) : ("/model/layers.0/attn/q_norm/Reshape_2/output_0_DequantizeLinear_Output": tensor(float),"/model/layers.0/attn/k_norm/Reshape_2/output_0_DequantizeLinear_Output": tensor(float),"/model/layers.0/attn/v_proj/MatMul/output_0_DequantizeLinear_Output": tensor(float),"past_key_values.0.key": tensor(float),"past_key_values.0.value": tensor(float),"past_seq_len": tensor(int32),"total_seq_len": tensor(int32),"cos_cache": tensor(float),"sin_cache": tensor(float),"","",) -> ("/model/layers.0/attn/GroupQueryAttention/output_0": tensor(float),"present.0.key": tensor(float),"present.0.value": tensor(float),) , Error Unrecognized attribute: qk_output for operator GroupQueryAttention

Not sure why this happen.

Maybe you could

1. Remove epcontextbinarygenerator pass to see if it could converted correctly and let winml / onnxruntime compile it when inferencing

2, Remove cache and run again

[nnbw-liu]

@xieofxie
epcontextbinarygenerator
Isn't this step a key operation to support QNN? Can this step be performed when onnxrruntime.qnn initializes the model?

[xieofxie]

yes, runtime could also do it for you.
you could save the ctx file in runtime via https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html#dump-qnn-context-binary

Or in WinML https://learn.microsoft.com/en-us/windows/ai/new-windows-ml/run-onnx-models?tabs=python#compile-models

Oh, for genai, I think it will compile the model for you, but I don't know if there is a way to save it

Maybe you could try to add it in the session_options in the genai_config.json

[xieofxie]

if you clear the cache and still see this error, maybe @jambayk could comment

[nnbw-liu]

if you clear the cache and still see this error, maybe @jambayk could comment

@xieofxie It's still the same problem to regenerate after clearing the cache

[nnbw-liu]

@xieofxie
Also, I have a question to ask.
My PC1 can be converted to the qwen3-4b model. But my PC2 doesn't work? Why is this? Here are the logs:

INFO  Saved Quantize Config: 
{
  "bits": 4,
  "group_size": -1,
  "desc_act": true,
  "sym": true,
  "lm_head": false,
  "quant_method": "gptq",
  "checkpoint_format": "gptq",
  "pack_dtype": "int32",
  "meta": {
    "quantizer": [
      "gptqmodel:4.0.0-dev"
    ],
    "uri": "https://github.com/modelcloud/gptqmodel",
    "damp_percent": 0.05,
    "damp_auto_increment": 0.01,
    "static_groups": false,
    "true_sequential": true,
    "mse": 0.0,
    "v2": false,
    "v2_alpha": 0.25
  }
}
Files in directory:
config.json
generation_config.json
quantize_config.json
quant_log.csv
Content of saved `generation_config.json`:
{
    "bos_token_id": 151643,
    "do_sample": true,
    "eos_token_id": [
        151645,
        151643
    ],
    "pad_token_id": 151643,
    "temperature": 0.7,
    "top_k": 20,
    "top_p": 0.8,
    "transformers_version": "4.53.2"
}
Content of saved `config.json`:
{
    "architectures": [
        "Qwen3ForCausalLM"
    ],
    "attention_bias": false,
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "head_dim": 128,
    "hidden_act": "silu",
    "hidden_size": 2560,
    "initializer_range": 0.02,
    "intermediate_size": 9728,
    "layer_types": [
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention",
        "full_attention"
    ],
    "max_position_embeddings": 262144,
    "max_window_layers": 36,
    "model_type": "qwen3",
    "num_attention_heads": 32,
    "num_hidden_layers": 36,
    "num_key_value_heads": 8,
    "quantization_config": {
        "bits": 4,
        "checkpoint_format": "gptq",
        "desc_act": true,
        "group_size": -1,
        "lm_head": false,
        "meta": {
            "damp_auto_increment": 0.01,
            "damp_percent": 0.05,
            "mse": 0.0,
            "quantizer": [
                "gptqmodel:4.0.0-dev"
            ],
            "static_groups": false,
            "true_sequential": true,
            "uri": "https://github.com/modelcloud/gptqmodel",
            "v2": false,
            "v2_alpha": 0.25
        },
        "pack_dtype": "int32",
        "quant_method": "gptq",
        "sym": true
    },
    "rms_norm_eps": 1e-06,
    "rope_scaling": null,
    "rope_theta": 5000000,
    "sliding_window": null,
    "tie_word_embeddings": false,
    "torch_dtype": "float32",
    "transformers_version": "4.53.2",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 151936
}
DEBUG Received safetensors_metadata: {'format': 'pt'}
INFO  Pre-Quantized model size: 8414.17MB, 8.22GB
INFO  Quantized model size: 4707.16MB, 4.60GB
INFO  Size difference: 3707.00MB, 3.62GB - 44.06%
[2025-10-14 12:25:04,247] [INFO] [engine.py:753:_run_pass] Pass g:gptqmodel finished in 1663.861205 seconds
[2025-10-14 12:25:04,253] [INFO] [engine.py:685:_run_pass] Running pass cs:capturesplitinfo
WARNING - AutoGPTQ has stopped development. Please transition to GPTQModel: https://github.com/ModelCoud/GPTQModel
GPTQModel has been merged into Transformers/Optimum and full deprecation of AutoGPTQ within HF frameworks is planned in the near-future.
WARNING:auto_gptq.nn_modules.qlinear.qlinear_cuda:CUDA extension not installed.
WARNING:auto_gptq.nn_modules.qlinear.qlinear_cuda_old:CUDA extension not installed.
Detected gptqmodel and auto-gptq, will use gptqmodel
INFO   Kernel: Auto-selection: adding candidate `TorchQuantLinear`
`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.25it/s]
INFO  Format: Converting `checkpoint_format` from `FORMAT.GPTQ` to internal `FORMAT.GPTQ_V2`.
INFO  Format: Converting GPTQ v1 to v2
INFO  Format: Conversion complete: 0.007999181747436523s
INFO  Optimize: `TorchQuantLinear` compilation triggered.
[2025-10-14 12:25:08,927] [INFO] [engine.py:753:_run_pass] Pass cs:capturesplitinfo finished in 4.673891 seconds
[2025-10-14 12:25:08,944] [INFO] [engine.py:685:_run_pass] Running pass mb:modelbuilder
[2025-10-14 12:25:09,240] [ERROR] [engine.py:748:_run_pass] Pass run failed.
Traceback (most recent call last):
  File "d:\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 736, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\systems\local.py", line 45, in run_pass
    output_model = the_pass.run(model, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\passes\olive_pass.py", line 242, in run
    output_model = self._run_for_config(model, self.config, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\passes\onnx\model_builder.py", line 256, in _run_for_config
    create_model(
  File "d:\olive\.venv\Lib\site-packages\onnxruntime_genai\models\builder.py", line 3402, in create_model
    raise NotImplementedError(f"The {hf_name} model is not currently supported.")
NotImplementedError: The D:\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN\cache\default_workflow\runs\8d8de7f5\models model is not currently supported.
[2025-10-14 12:25:09,326] [WARNING] [engine.py:310:run_accelerator] Failed to run Olive on npu-qnn.
Traceback (most recent call last):
  File "d:\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 306, in run_accelerator
    output_footprint = self._run_no_search(input_model_config, input_model_id, accelerator_spec, output_dir)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 350, in _run_no_search
    should_prune, signal, model_ids = self._run_passes(input_model_config, input_model_id, accelerator_spec)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 641, in _run_passes
    model_config, model_id = self._run_pass(
                             ^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\engine\engine.py", line 736, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\systems\local.py", line 45, in run_pass
    output_model = the_pass.run(model, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\passes\olive_pass.py", line 242, in run
    output_model = self._run_for_config(model, self.config, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\olive\.venv\Lib\site-packages\olive\passes\onnx\model_builder.py", line 256, in _run_for_config
    create_model(
  File "d:\olive\.venv\Lib\site-packages\onnxruntime_genai\models\builder.py", line 3402, in create_model
    raise NotImplementedError(f"The {hf_name} model is not currently supported.")
NotImplementedError: The D:\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN\cache\default_workflow\runs\8d8de7f5\models model is not currently supported.
[2025-10-14 12:25:09,331] [INFO] [engine.py:235:run] Run history for npu-qnn:
[2025-10-14 12:25:09,347] [INFO] [engine.py:499:dump_run_history] run history:
+------------+-------------------+------------------+----------------+-----------+
| model_id   | parent_model_id   | from_pass        |   duration_sec | metrics   |
+============+===================+==================+================+===========+
| d4993402   |                   |                  |                |           |
+------------+-------------------+------------------+----------------+-----------+
| d18e18b5   | d4993402          | quarot           |       98.0305  |           |
+------------+-------------------+------------------+----------------+-----------+
| 8d8de7f5   | d18e18b5          | gptqmodel        |     1663.86    |           |
+------------+-------------------+------------------+----------------+-----------+
| 0cd3e38e   | 8d8de7f5          | capturesplitinfo |        4.67389 |           |
+------------+-------------------+------------------+----------------+-----------+
[2025-10-14 12:25:09,348] [WARNING] [engine.py:259:run] No output model
No output model produced. Please check the log for details.

Computer Configuration 1:​
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (3.70 GHz)​
RAM: 32GB​
GPU: NVIDIA GeForce GTX 1080 Ti 11G​
CUDA: 12.1​

Computer Configuration 2:​
CPU: Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz (2.10 GHz)​
RAM: 256GB​
GPU: NVIDIA GeForce RTX 4090 24G​
CUDA: 12.8

[nnbw-liu]

~ yes, runtime could also do it for you. you could save the ctx file in runtime via https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html#dump-qnn-context-binary ~ Or in WinML https://learn.microsoft.com/en-us/windows/ai/new-windows-ml/run-onnx-models?tabs=python#compile-models~~

Oh, for genai, I think it will compile the model for you, but I don't know if there is a way to save it

Maybe you could try to add it in the in the session_options``genai_config.json

@xieofxie
This is not possible. I tried to initialize the model using GenAI.QNN and it was successfully initialized. However, during inference, only two characters were output and it stopped, but the inference did not terminate. The transformed model still seems to have issues.

[xieofxie]

this NotImplementedError: The D:\olive\olive-recipes\Qwen-Qwen2.5-7B-Instruct\QNN\cache\default_workflow\runs\8d8de7f5\models model is not currently supported. looks like a bug in olive

let's wait someone from olive team to look into your issue..