Added fallback to preload cudnn dlls from nvidia cudnn venv package or torch venv package#1119
Added fallback to preload cudnn dlls from nvidia cudnn venv package or torch venv package#1119hthadicherla wants to merge 1 commit intomainfrom
Conversation
📝 WalkthroughWalkthroughAdded a fallback in Changes
Sequence Diagram(s)sequenceDiagram
participant Caller
participant _check_for_libcudnn
participant OS_Loader as "OS library search (PATH/LD_LIBRARY_PATH)"
participant ONNXRT as "onnxruntime.preload_dlls()"
Caller->>_check_for_libcudnn: invoke
_check_for_libcudnn->>OS_Loader: search for libcudnn
alt found
OS_Loader-->>_check_for_libcudnn: path to libcudnn
_check_for_libcudnn-->>Caller: return True
else not found
OS_Loader-->>_check_for_libcudnn: not found
_check_for_libcudnn->>ONNXRT: attempt preload_dlls()
alt preload succeeds
ONNXRT-->>_check_for_libcudnn: success
_check_for_libcudnn-->>Caller: return True
else preload fails or raises
ONNXRT-->>_check_for_libcudnn: exception / failure
_check_for_libcudnn-->>Caller: raise FileNotFoundError (mentions env var + preload guidance)
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@modelopt/onnx/quantization/ort_utils.py`:
- Around line 76-93: The error text incorrectly states that
onnxruntime.preload_dlls() failed even when preload_dlls doesn't exist; update
the failure path in the block around ort.preload_dlls to distinguish two cases:
if hasattr(ort, "preload_dlls") is False, log/error that preload_dlls is
unavailable in this ORT version (mention ort.preload_dlls is not present) and
include env_variable/lib_pattern in the message; if hasattr is True but the
try/except catches an exception from ort.preload_dlls(), preserve the existing
warning and then raise the FileNotFoundError with a message that clearly says
preload_dlls was attempted and failed (referencing ort.preload_dlls,
logger.warning, logger.error, env_variable, lib_pattern, and the final
FileNotFoundError).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 26fd6dd9-298b-47d5-bb9c-a56c9bebd6c5
📒 Files selected for processing (1)
modelopt/onnx/quantization/ort_utils.py
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1119 +/- ##
==========================================
+ Coverage 70.18% 70.21% +0.02%
==========================================
Files 228 228
Lines 25952 25960 +8
==========================================
+ Hits 18215 18227 +12
+ Misses 7737 7733 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…a-cudnn-cu12 package incase the dlls don't exist in system path Signed-off-by: Hrishith Thadicherla <hthadicherla@nvidia.com>
hthadicherla
force-pushed
the
hthadicherla/add-ort-preload-dll-cudnn
branch
from
56f5c33 to
c9f7052
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@modelopt/onnx/quantization/ort_utils.py`:
- Around line 77-86: preload_dlls() can silently do nothing, so update the
preload check in ort_utils.py to call ort.preload_dlls(verbose=True) and then
verify the runtime actually loaded GPU providers by inspecting
ort.get_all_providers() (check for 'CUDAExecutionProvider' and/or
'CUDNNExecutionProvider'); only return True when those providers are present,
otherwise log a warning with the provider list and return False so we don't
report success when DLLs weren't loaded.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ce2ec865-46e8-45b6-b906-3214ad906a93
📒 Files selected for processing (1)
modelopt/onnx/quantization/ort_utils.py
| try: | ||
| ort.preload_dlls() | ||
| logger.info( | ||
| "onnxruntime.preload_dlls() succeeded; CUDA/cuDNN DLLs preloaded from site-packages." | ||
| " Please check that this is the correct version needed for your ORT version at" | ||
| " https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements." | ||
| ) | ||
| return True | ||
| except Exception as e: | ||
| logger.warning(f"onnxruntime.preload_dlls() also failed: {e}") |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
onnxruntime preload_dlls return value documentation
💡 Result:
onnxruntime.preload_dlls is a Python function available in onnxruntime-gpu package since version 1.21.0. It preloads CUDA, cuDNN, and MSVC runtime DLLs (on Windows) or shared libraries (on Linux) to ensure compatibility, especially with PyTorch. Function signature: onnxruntime.preload_dlls(cuda=True, cudnn=True, msvc=True, directory=None, verbose=False) It returns None (void function). No return value is documented or shown in usage examples across official docs, PRs, and source snippets. Calling it performs side effects (loading DLLs) and may print loaded DLLs if verbose=True, but does not return any value. Official documentation: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#preload-dlls (result 4 from first search). Source introduced in PR #23674: microsoft/onnxruntime#23674 (multiple results). Example usage: import onnxruntime onnxruntime.preload_dlls(verbose=True) # Prints list of loaded DLLs if successful, returns None
Citations:
- 1: [CUDA] Preload dependent DLLs microsoft/onnxruntime#23674
- 2: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html
- 3: https://gitlab.ispras.ru/mvg/mvg-oss/onnxruntime/-/blob/v1.23.2/onnxruntime/__init__.py?ref_type=tags
preload_dlls() may silently fail without raising an exception, causing incorrect success reporting.
According to the official ONNX Runtime documentation, preload_dlls() is a void function that returns None and does not raise exceptions when DLLs are missing — it fails gracefully and continues. With the default parameters used here (no verbose=True), missing cuDNN will produce no warning or error, causing the function to return True even when the library was never loaded.
Either call preload_dlls(verbose=True) to detect missing libraries in the logging output, validate that the libraries were actually loaded after the call, or remove this unreliable check entirely.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@modelopt/onnx/quantization/ort_utils.py` around lines 77 - 86, preload_dlls()
can silently do nothing, so update the preload check in ort_utils.py to call
ort.preload_dlls(verbose=True) and then verify the runtime actually loaded GPU
providers by inspecting ort.get_all_providers() (check for
'CUDAExecutionProvider' and/or 'CUDNNExecutionProvider'); only return True when
those providers are present, otherwise log a warning with the provider list and
return False so we don't report success when DLLs weren't loaded.
1 participant
What does this PR do?
Type of change: Bug fix
There was a QA team that was testing the modelopt 0.43 release and pointed out that we could install nvidia-cudnn pypi packages and use ort.preload_dlls() to load the dlls from the python venv instead of trying to search in system path only .
Here is the info about onnxruntime.preload_dlls() function
So added fallback to system path cudnn search to preload dlls and if that also fails then raise exception.
Testing
Tested quantization by installing nvidia-cudnn-cu12 package and removing cudnn dlls from system path. Working as expected.
Summary by CodeRabbit