Skip to content

Attenion(23) CUDA #7643

Attenion(23) CUDA

Attenion(23) CUDA #7643

Triggered via pull request November 19, 2025 03:38
Status Failure
Total duration 23m 44s
Artifacts

linux_tensorrt_ci.yml

on: pull_request
Build Linux TensorRT x64 Release  /  build_test_pipeline
14m 1s
Build Linux TensorRT x64 Release / build_test_pipeline
Test Linux TensorRT x64 Release
0s
Test Linux TensorRT x64 Release
Fit to window
Zoom out
Zoom in

Annotations

4 errors and 3 warnings
Build Linux TensorRT x64 Release / build_test_pipeline
Action failed with error: Command "docker run --rm --volume /mnt/vss/_work/onnxruntime/onnxruntime:/onnxruntime_src --volume /mnt/vss/_work/_temp:/onnxruntime_src/build --volume /home/cloudtest/.cache:/home/onnxruntimedev/.cache -w /onnxruntime_src -e ALLOW_RELEASED_ONNX_OPSET_ONLY=0 -e NIGHTLY_BUILD=1 -e RUNNER_TEMP=/onnxruntime_src/build ghcr.io/microsoft/onnxruntime/onnxruntimetensorrt86gpubuild:sha256-c57931701dc717a81351f6311d0b1e8fc89d0aea1f184b0324011e87fe6039b8 /bin/bash -c set -ex; PATH=/opt/python/cp310-cp310/bin:$PATH && python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt && python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build" failed with exit code 1
Build Linux TensorRT x64 Release / build_test_pipeline
stderr: + PATH=/opt/python/cp310-cp310/bin:/usr/local/dotnet:/opt/rh/gcc-toolset-14/root/usr/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/lib/jvm/msopenjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt WARNING: The script isympy is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script pygmentize is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts f2py and numpy-config are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts dmypy, mypy, mypyc, stubgen and stubtest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts py.test and pytest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts backend-test-tools, check-model and check-node are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. [notice] A new release of pip is available: 25.1.1 -> 25.3 [notice] To update, run: pip install --upgrade pip + python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build 2025-11-19 03:57:57,723 build [DEBUG] - Command line arguments: --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build 2025-11-19 03:57:57,726 build [INFO] - Build started 2025-11-19 03:57:57,726 build [INFO] - Building targets for Release configuration 2025-11-19 03:57:57,727 build [INFO] - /usr/bin/cmake --build build/Release/Release --config Release -- -j16 Traceback (most recent call last): File "/onnxruntime_src/tools/ci_build/build.py", line 2714, in <module> sys.exit(main()) File "/onnxruntime_src/tools/ci_build/build.py", line 2614, in main build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.targets) File "/onnxruntime_src/tools/ci_build/build.py", line 1376, in build_targets run_subprocess(cmd_args, env=env) File "/onnxruntime_src/tools/ci_build/build.py", line 148, in run_subprocess return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env) File "/onnxruntime_src/tools/python/util/run.py", line 50, in run completed_process = subprocess.run( File "/opt/python/cp310-cp310/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessE
Build Linux TensorRT x64 Release / build_test_pipeline
stdout: Ignoring numpy: markers 'python_version >= "3.14"' don't match your environment Ignoring protobuf: markers 'python_version >= "3.14"' don't match your environment Collecting numpy==2.2.6 (from -r tools/ci_build/github/linux/python/requirements.txt (line 1)) Using cached numpy-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB) Collecting mypy (from -r tools/ci_build/github/linux/python/requirements.txt (line 3)) Using cached mypy-1.18.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.2 kB) Collecting pytest (from -r tools/ci_build/github/linux/python/requirements.txt (line 4)) Using cached pytest-9.0.1-py3-none-any.whl.metadata (7.6 kB) Requirement already satisfied: setuptools>=68.2.2 in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 5)) (80.9.0) Requirement already satisfied: wheel in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 6)) (0.45.1) Collecting protobuf==4.25.1 (from -r tools/ci_build/github/linux/python/requirements.txt (line 7)) Using cached protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes) Collecting sympy==1.14 (from -r tools/ci_build/github/linux/python/requirements.txt (line 9)) Using cached sympy-1.14.0-py3-none-any.whl.metadata (12 kB) Collecting flatbuffers (from -r tools/ci_build/github/linux/python/requirements.txt (line 10)) Using cached flatbuffers-25.9.23-py2.py3-none-any.whl.metadata (875 bytes) Collecting psutil (from -r tools/ci_build/github/linux/python/requirements.txt (line 11)) Using cached psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl.metadata (23 kB) Collecting onnxscript==0.5.3 (from -r tools/ci_build/github/linux/python/requirements.txt (line 12)) Using cached onnxscript-0.5.3-py3-none-any.whl.metadata (13 kB) Collecting onnx-ir==0.1.10 (from -r tools/ci_build/github/linux/python/requirements.txt (line 13)) Using cached onnx_ir-0.1.10-py3-none-any.whl.metadata (3.6 kB) Collecting jinja2 (from -r tools/ci_build/github/linux/python/requirements.txt (line 14)) Using cached jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB) Collecting markupsafe (from -r tools/ci_build/github/linux/python/requirements.txt (line 15)) Using cached markupsafe-3.0.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB) Collecting onnx==1.19.1 (from -r tools/ci_build/github/linux/python/requirements.txt (line 16)) Using cached onnx-1.19.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (7.0 kB) Collecting mpmath<1.4,>=1.1.0 (from sympy==1.14->-r tools/ci_build/github/linux/python/requirements.txt (line 9)) Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB) Collecting ml_dtypes (from onnxscript==0.5.3->-r tools/ci_build/github/linux/python/requirements.txt (line 12)) Using cached ml_dtypes-0.5.4-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (8.9 kB) Requirement already satisfied: packaging in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from onnxscript==0.5.3->-r tools/ci_build/github/linux/python/requirements.txt (line 12)) (25.0) Collecting typing_extensions>=4.10 (from onnxscript==0.5.3->-r tools/ci_build/github/linux/python/requirements.txt (line 12)) Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) Collecting mypy_extensions>=1.0.0 (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 3)) Using cached mypy_extensions-1.1.0-py3-none-any.whl.metadata (1.1 kB) Collecting pathspec>=0.9.0 (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 3)) Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB) Requirement already satisfied: tomli>=1.1.0 in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 3)) (2.2.1) Collect
Build Linux TensorRT x64 Release / build_test_pipeline
Command failed with exit code 1
Build Linux TensorRT x64 Release / build_test_pipeline
Wheel output directory /mnt/vss/_work/_temp/Release/dist does not exist.
Build Linux TensorRT x64 Release / build_test_pipeline
stderr: + PATH=/opt/python/cp310-cp310/bin:/usr/local/dotnet:/opt/rh/gcc-toolset-14/root/usr/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/lib/jvm/msopenjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt WARNING: The script isympy is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script pygmentize is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts f2py and numpy-config are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts dmypy, mypy, mypyc, stubgen and stubtest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts py.test and pytest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts backend-test-tools, check-model and check-node are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. [notice] A new release of pip is available: 25.1.1 -> 25.3 [notice] To update, run: pip install --upgrade pip + python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --update 2025-11-19 03:56:44,447 build [DEBUG] - Command line arguments: --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --update 2025-11-19 03:56:44,450 build [INFO] - Build started 2025-11-19 03:56:44,450 build [INFO] - Generating CMake build tree 2025-11-19 03:56:44,469 build [INFO] - /usr/bin/cmake /onnxruntime_src/cmake -Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF -Donnxruntime_RUN_ONNX_TESTS=ON -Donnxruntime_GENERATE_TEST_REPORTS=ON -DPython_EXECUTABLE=/opt/python/cp310-cp310/bin/python3 -Donnxruntime_USE_VCPKG=ON -Donnxruntime_USE_MIMALLOC=OFF -Donnxruntime_ENABLE_PYTHON=ON -Donnxruntime_BUILD_CSHARP=OFF -Donnxruntime_BUILD_JAVA=ON -Donnxruntime_BUILD_NODEJS=OFF -Donnxruntime_BUILD_OBJC=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF -Donnxruntime_USE_DNNL=OFF -Donnxruntime_USE_NNAPI_BUILTIN=OFF -Donnxruntime_USE_VSINPU=OFF -Donnxruntime_USE_RKNPU=OFF -Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF -Donnxruntime_USE_VITISAI=OFF -Donnxruntime_USE_TENSORRT=ON -Donnxruntime_USE_NV=OFF -Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON -Donnxruntime_USE_TENSORRT_INTERFACE=OFF -Donnxruntime_USE_CUDA_INTERFACE=OFF -Donnxruntime_USE_NV_INTERFACE=OFF -Donnxruntime_
Build Linux TensorRT x64 Release / build_test_pipeline
stderr: WARNING! Your credentials are stored unencrypted in '/home/cloudtest/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/