Add dx_shm_size and dx_ipc_mode WDL runtime hints#532
Open
bomquangia wants to merge 1 commit intodevelopfrom
Open
Add dx_shm_size and dx_ipc_mode WDL runtime hints#532bomquangia wants to merge 1 commit intodevelopfrom
bomquangia wants to merge 1 commit intodevelopfrom
Conversation
New opt-in runtime hints allow WDL tasks to configure Docker's shared memory size and IPC namespace mode, which are needed for workloads that require large /dev/shm allocations (e.g. distributed deep learning with collective communication libraries). Both hints are optional and validated against strict allowlists to prevent shell-meta injection into the rendered docker run command. Depends on wdlTools 0.17.18 release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dx_shm_sizeanddx_ipc_mode--shm-sizeand--ipcflags in the generateddocker runcommandUsage
WDL 1.x (
runtimeblock):WDL 2.0 (
hintsblock):Both values are WDL expressions and can be overridden per-invocation via
overrides___job inputs.Changes
core/src/main/scala/dx/core/languages/wdl/Runtime.scala—ShmSize/IpcModeDxRuntimeHints, validation, accessorsexecutorWdl/src/main/scala/dx/executor/wdl/WdlTaskExecutor.scala— threads hints intogenerator.apply(...)core/src/test/scala/dx/core/languages/wdl/RuntimeHintsTest.scala— 11 new testsdoc/ExpertOptions.md— user-facing documentationRELEASE_NOTES.md— changelog entrybuild.sbt— bumpswdlToolsdependency to 0.17.18Dependencies
Requires wdlTools 0.17.18 to be released before this PR's CI will pass dependency resolution.
Test plan
containerRunScriptcontains--shm-size/--ipcwhen hints are set