Attenion(23) CUDA #7643
Triggered via pull request
November 19, 2025 03:38
Status
Failure
Total duration
23m 44s
Artifacts
–
linux_tensorrt_ci.yml
on: pull_request
Build Linux TensorRT x64 Release
/
build_test_pipeline
14m 1s
Test Linux TensorRT x64 Release
0s
Annotations
4 errors and 3 warnings
|
Build Linux TensorRT x64 Release / build_test_pipeline
Action failed with error: Command "docker run --rm --volume /mnt/vss/_work/onnxruntime/onnxruntime:/onnxruntime_src --volume /mnt/vss/_work/_temp:/onnxruntime_src/build --volume /home/cloudtest/.cache:/home/onnxruntimedev/.cache -w /onnxruntime_src -e ALLOW_RELEASED_ONNX_OPSET_ONLY=0 -e NIGHTLY_BUILD=1 -e RUNNER_TEMP=/onnxruntime_src/build ghcr.io/microsoft/onnxruntime/onnxruntimetensorrt86gpubuild:sha256-c57931701dc717a81351f6311d0b1e8fc89d0aea1f184b0324011e87fe6039b8 /bin/bash -c set -ex; PATH=/opt/python/cp310-cp310/bin:$PATH && python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt && python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build" failed with exit code 1
|
|
Build Linux TensorRT x64 Release / build_test_pipeline
stderr: + PATH=/opt/python/cp310-cp310/bin:/usr/local/dotnet:/opt/rh/gcc-toolset-14/root/usr/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/lib/jvm/msopenjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt
WARNING: The script isympy is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script pygmentize is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts f2py and numpy-config are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts dmypy, mypy, mypyc, stubgen and stubtest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts py.test and pytest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts backend-test-tools, check-model and check-node are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
[notice] A new release of pip is available: 25.1.1 -> 25.3
[notice] To update, run: pip install --upgrade pip
+ python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build
2025-11-19 03:57:57,723 build [DEBUG] - Command line arguments:
--build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build
2025-11-19 03:57:57,726 build [INFO] - Build started
2025-11-19 03:57:57,726 build [INFO] - Building targets for Release configuration
2025-11-19 03:57:57,727 build [INFO] - /usr/bin/cmake --build build/Release/Release --config Release -- -j16
Traceback (most recent call last):
File "/onnxruntime_src/tools/ci_build/build.py", line 2714, in <module>
sys.exit(main())
File "/onnxruntime_src/tools/ci_build/build.py", line 2614, in main
build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.targets)
File "/onnxruntime_src/tools/ci_build/build.py", line 1376, in build_targets
run_subprocess(cmd_args, env=env)
File "/onnxruntime_src/tools/ci_build/build.py", line 148, in run_subprocess
return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
File "/onnxruntime_src/tools/python/util/run.py", line 50, in run
completed_process = subprocess.run(
File "/opt/python/cp310-cp310/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessE
|
|
Build Linux TensorRT x64 Release / build_test_pipeline
stdout: Ignoring numpy: markers 'python_version >= "3.14"' don't match your environment
Ignoring protobuf: markers 'python_version >= "3.14"' don't match your environment
Collecting numpy==2.2.6 (from -r tools/ci_build/github/linux/python/requirements.txt (line 1))
Using cached numpy-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
Collecting mypy (from -r tools/ci_build/github/linux/python/requirements.txt (line 3))
Using cached mypy-1.18.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.2 kB)
Collecting pytest (from -r tools/ci_build/github/linux/python/requirements.txt (line 4))
Using cached pytest-9.0.1-py3-none-any.whl.metadata (7.6 kB)
Requirement already satisfied: setuptools>=68.2.2 in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 5)) (80.9.0)
Requirement already satisfied: wheel in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 6)) (0.45.1)
Collecting protobuf==4.25.1 (from -r tools/ci_build/github/linux/python/requirements.txt (line 7))
Using cached protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting sympy==1.14 (from -r tools/ci_build/github/linux/python/requirements.txt (line 9))
Using cached sympy-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting flatbuffers (from -r tools/ci_build/github/linux/python/requirements.txt (line 10))
Using cached flatbuffers-25.9.23-py2.py3-none-any.whl.metadata (875 bytes)
Collecting psutil (from -r tools/ci_build/github/linux/python/requirements.txt (line 11))
Using cached psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl.metadata (23 kB)
Collecting onnxscript==0.5.3 (from -r tools/ci_build/github/linux/python/requirements.txt (line 12))
Using cached onnxscript-0.5.3-py3-none-any.whl.metadata (13 kB)
Collecting onnx-ir==0.1.10 (from -r tools/ci_build/github/linux/python/requirements.txt (line 13))
Using cached onnx_ir-0.1.10-py3-none-any.whl.metadata (3.6 kB)
Collecting jinja2 (from -r tools/ci_build/github/linux/python/requirements.txt (line 14))
Using cached jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting markupsafe (from -r tools/ci_build/github/linux/python/requirements.txt (line 15))
Using cached markupsafe-3.0.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB)
Collecting onnx==1.19.1 (from -r tools/ci_build/github/linux/python/requirements.txt (line 16))
Using cached onnx-1.19.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (7.0 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy==1.14->-r tools/ci_build/github/linux/python/requirements.txt (line 9))
Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting ml_dtypes (from onnxscript==0.5.3->-r tools/ci_build/github/linux/python/requirements.txt (line 12))
Using cached ml_dtypes-0.5.4-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (8.9 kB)
Requirement already satisfied: packaging in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from onnxscript==0.5.3->-r tools/ci_build/github/linux/python/requirements.txt (line 12)) (25.0)
Collecting typing_extensions>=4.10 (from onnxscript==0.5.3->-r tools/ci_build/github/linux/python/requirements.txt (line 12))
Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting mypy_extensions>=1.0.0 (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 3))
Using cached mypy_extensions-1.1.0-py3-none-any.whl.metadata (1.1 kB)
Collecting pathspec>=0.9.0 (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 3))
Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: tomli>=1.1.0 in /opt/_internal/cpython-3.10.18/lib/python3.10/site-packages (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 3)) (2.2.1)
Collect
|
|
Build Linux TensorRT x64 Release / build_test_pipeline
Command failed with exit code 1
|
|
Build Linux TensorRT x64 Release / build_test_pipeline
Wheel output directory /mnt/vss/_work/_temp/Release/dist does not exist.
|
|
Build Linux TensorRT x64 Release / build_test_pipeline
stderr: + PATH=/opt/python/cp310-cp310/bin:/usr/local/dotnet:/opt/rh/gcc-toolset-14/root/usr/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/lib/jvm/msopenjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt
WARNING: The script isympy is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script pygmentize is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts f2py and numpy-config are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts dmypy, mypy, mypyc, stubgen and stubtest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts py.test and pytest are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts backend-test-tools, check-model and check-node are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
[notice] A new release of pip is available: 25.1.1 -> 25.3
[notice] To update, run: pip install --upgrade pip
+ python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --update
2025-11-19 03:56:44,447 build [DEBUG] - Command line arguments:
--build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.8 --cuda_home=/usr/local/cuda-12.8 --cudnn_home=/usr/local/cuda-12.8 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --update
2025-11-19 03:56:44,450 build [INFO] - Build started
2025-11-19 03:56:44,450 build [INFO] - Generating CMake build tree
2025-11-19 03:56:44,469 build [INFO] - /usr/bin/cmake /onnxruntime_src/cmake -Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF -Donnxruntime_RUN_ONNX_TESTS=ON -Donnxruntime_GENERATE_TEST_REPORTS=ON -DPython_EXECUTABLE=/opt/python/cp310-cp310/bin/python3 -Donnxruntime_USE_VCPKG=ON -Donnxruntime_USE_MIMALLOC=OFF -Donnxruntime_ENABLE_PYTHON=ON -Donnxruntime_BUILD_CSHARP=OFF -Donnxruntime_BUILD_JAVA=ON -Donnxruntime_BUILD_NODEJS=OFF -Donnxruntime_BUILD_OBJC=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF -Donnxruntime_USE_DNNL=OFF -Donnxruntime_USE_NNAPI_BUILTIN=OFF -Donnxruntime_USE_VSINPU=OFF -Donnxruntime_USE_RKNPU=OFF -Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF -Donnxruntime_USE_VITISAI=OFF -Donnxruntime_USE_TENSORRT=ON -Donnxruntime_USE_NV=OFF -Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON -Donnxruntime_USE_TENSORRT_INTERFACE=OFF -Donnxruntime_USE_CUDA_INTERFACE=OFF -Donnxruntime_USE_NV_INTERFACE=OFF -Donnxruntime_
|
|
Build Linux TensorRT x64 Release / build_test_pipeline
stderr: WARNING! Your credentials are stored unencrypted in '/home/cloudtest/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
|