-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Update weight sharing tool to support plugin EPs #26614
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Update weight sharing tool to support plugin EPs #26614
Conversation
| # ONNXRuntime EP Context Model Generation with Weight Sharing | ||
|
|
||
| > [!NOTE] | ||
| > This tool is deprecated. Please use the public ONNX Runtime Python APIs to compile models with resource sharing. Refer to the example Python script at the end of this document. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm thinking we remove this tool after ORT 1.24. At that point we can consider replacing it with a Python version if we deem it necessary (although a simple example script in the docs is probably enough).
| Ort::Env env(logging_level, "ep_weight_sharing"); | ||
|
|
||
| ORT_TRY { | ||
| std::optional<PluginEpLibraryRegistrationHandle> plugin_ep_library_registration_handle = std::nullopt; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: optional<unique_ptr> feels unnecessary. can we just use an empty unique_ptr to represent the empty state?
| Test compiling two example models using weight sharing (via example plugin EP) | ||
| """ | ||
| if sys.platform != "win32": | ||
| self.skipTest("Skipping test because device discovery is only supported on Windows") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there should be some non-Windows support for device discovery now
Description
ep_weight_sharing_ctx_gentool to support specifying a plugin EP configuration (via JSON).ep_weight_sharing_ctx_gentool as deprecated and add notification to README that recommends the use the public Python ORT APIs instead.Tool usage
Create a JSON file that contains information about the plugin EP to load/use (e.g.,
example_plugin_ep_config.json):{ "ep_library_registration_name": "example_plugin_ep", "ep_library_path": "example_plugin_ep.dll", "selected_ep_name": "example_plugin_ep", "default_ep_options": { "option_key": "option_value" } }Call the
ep_weight_sharing_ctx_gentool with the-pcommand-line option to specify the location of the above configuration file:$ ep_weight_sharing_ctx_gen.exe -p example_plugin_ep_config.json model_1.onnx,model_2.onnxMotivation and Context
Close the functionality gap between traditional provider-bridge EPs and plugin EPs. This PR allows using plugin EPs with the tool that compiles models with weight sharing.