Skip to content

Passing both reference and value to job#870

Open
gpetretto wants to merge 1 commit intomaterialsproject:mainfrom
gpetretto:resolved_reference
Open

Passing both reference and value to job#870
gpetretto wants to merge 1 commit intomaterialsproject:mainfrom
gpetretto:resolved_reference

Conversation

@gpetretto
Copy link
Copy Markdown
Contributor

This PR introduces the option to have both the OutputReference and the corresponding value as job inputs when coming from the output of another job. This would be activated by setting resolve_references="both" in JobConfig.

One example use case where this may be useful: imagine to have multiple jobs that produce an output and having a final job that compares the outputs and needs to select one or more among them. Currently the only option would be to pass the values and store the selected value(s) as output of the final job. This has the downsides that 1) the data in the DB is duplicated 2) The reference to Job executing the calculation is lost.
With the option to pass "both" it would instead be possible to make the comparison while using the reference of the selected job as output.

A few notes on the implementation:

  • ResolvedReference does not contain explicitly the OutputReference because otherwise in jobflow-remote it gets re-resolved at runtime.
  • When resolve_references="both" the ResolvedReference is always used, even with OnMissing.None or OnMissing.PASS.
  • If resolve_references="both" is used in combination with OnMissing.PASS it can still cause problems for jobflow-remote. However, I have discovered that OnMissing.PASS is acutally not supported in jobflow-remote (I will address this). Is OnMissing.PASS is used somewhere?
  • I had considered an entirely different option for implementing this functionality: resolve_references could be a list of the variable names that need to be resolved. One can then solve the previous example by passing both the list of values and list of references. While this could have given more flexibility, the implementation is trickier and I believe less reliable (e.g. when the inputs are *args and **kwargs). This would also require passing the same reference twice in the example above, which does not seem very intuitive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant