-
Notifications
You must be signed in to change notification settings - Fork 30
CEP: Jinja functionality in v1 recipes #71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 6 commits
f1ad3f2
07da5e7
e4e0a42
81bdf68
2694476
62ea3d4
0d3f301
dde40b1
8900686
7bd813f
0fda213
c9cf442
9f2276a
16a7473
521ca47
a3f913a
07328a0
c1506fb
9b5e23d
fa7dbdb
c3c03c7
fba70b6
c102814
81d9515
822558a
4141b17
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,125 @@ | ||||||
| # Jinja functions in recipes | ||||||
|
|
||||||
| The new recipe format has some Jinja functionalities. We want to specify what functions exist and their expected behavior. | ||||||
|
||||||
| The new recipe format has some Jinja functionalities. We want to specify what functions exist and their expected behavior. | |
| The new recipe format (introduced by CEP-XX, CEP-XY) has some Jinja functionalities. We want to specify what functions exist and their expected behavior. |
wolfv marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
wolfv marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
wolfv marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
wolfv marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems a bit out of place; I think the example variant config should be provided before using its values to evaluate the function.
Additionally, we should provide an example variant config showing how to select different compilers for different platforms; e.g., are we keeping the conda_build_config.yaml mechanism?
c_compiler:
 - gcc # [linux]
- clang # [osx]
- vc # [win]
c_compiler_version:
 - "14.1" # [linux]
- "17.0" # [osx]
- "2022" # [win]There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah we need to write a spec for the variant config (or even more global "config"). But IMO that shouldn't be part of this CEP.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My comment disappeared, so re-adding it:
Does "centos" have to be mentioned here? CDT packages could technically contain binaries from any Linux distribution right? I'm also wondering if we should add a link to some documentation that describe CDT more in details?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's the current default value in conda-build. idk what to do, i am basically just documenting what's already in conda-build.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel a little bit uneasy with having default values hardcoded here. Do we really need to define default values in the spec? I feel like this kind of thing should be explicitly defined in an ecosystem (the central variant config in an ecosystem) instead of relying on some defaults.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that we should NOT have default values for CDTs. The choice of CentOS 6 and 7 was made by Anaconda and conda-forge long ago, but I don't think we should necessarily impose such choices on the ecosystem as a whole.
Further, imposing these defaults means we either have to indefinitely support CDTs from EOL Linux distributions (CentOS 6 hit EOL in 2020-Nov, and CentOS 7 will hit EOL in 2024-Jun), or the evaluation of ${{ cdt("...") }} changes depending on the version of {conda,rattler)-build you use. Neither of those outcomes seems ideal.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, we can get rid of this. I think we adopted this from conda-build where these values are also there by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW, in the new CDTs for Alma 8, we're planning to get rid of cdt_name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd also leave/move these out (see comment regarding defaults for compilers above).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@h-vetinari - curious what that means exactly? A change to conda-build?
I'll remove the defaults from the CEP!
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pin_subpackage is defined in another section. This section seems to only define the arguments of pin_subpackage and pin_compatible?
wolfv marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There seems a strong but unstated assumption here that version strings are integers separated by dots (.). If that is the case, that assumption should be explicitly stated and guidance provided as to what should happen if the assumption is violated. (Just sayin', because 1!24.4.0+91_gbfa8ccdce is a perfectly valid PEP-440/conda package version string.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, as written, it seems like min_pin and max_pin do not support an explicit version; e.g., recipes cannot simply have max_pin="1.2.3".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, as written, it seems like
min_pinandmax_pindo not support an explicit version; e.g., recipes cannot simply havemax_pin="1.2.3".
Conda's existing pin_compatible jinja has both {min,max}_pin as well as {lower,upper}_bound, in order to be able to distinguish 'x.x' from an explicit version limit.
Perhaps it's easier to understand to overload both styles into two kwargs (one for lower, one for upper), and allow either 'x(.x)*' or an explicit version number? It's more work to implement, but less confusing IMO (otherwise it's easy to specify conflicting things e.g. by setting both max_pin and upper_bound)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, I wasn't aware! I found a few places where it's used but I don't know how it works. For example, in the onnx-feedstock it says:
- {{ pin_compatible('numpy', lower_bound='1.19', upper_bound='3.0') }} # [py>38]
Does this still take the lower bound from the build-time resolution? Or is this equivalent to just writing numpy >=1.19,<3.0?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also don't really understand why we would want this in the function. One can just as well write something like:
- numpy >=1.5
- ${{ pin_compatible("numpy", max_pin="x.x", min_pin="x.x.x") }}
If one needs additional (hard-coded) constraints. The solver doesn't really care about more constraints.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@h-vetinari I'm on board with your proposal.
I suppose that could in theory create a problem if some upstream project decided to the string literals x.x, x.y, and x.z as their versions, but if that ever happens, I'm happy to say "NOPE" to attempting to package it. 😆
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wolfv, given your changes in 0fda213, I presume you're not on board with this idea (or didn't see this thread)?
If you want to keep {min,max}_pin separate from {lower,upper}_bound, I think we need to enforce that at most one {min_pin,lower_bound} resp. {max_pin,upper_bound} each may be specified, everything else should raise an error.
I'd still prefer a unified interface, because I don't see the gain to keep these arguments separate (but rather requires a lot of complexity to ensure it is unambiguous/consistent)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I indeed didn't see it or didn't read carefully enough. I also think this is a good change. The logic is already in place, so I don't think it will be a big change on the rattler-build side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I just implemented that we only use lower_bound and upper_bound, but now I am wondering if we shouldn't have used min_pin and max_pin instead (as more people are using that these days). But this PR has the implementation: prefix-dev/rattler-build#918
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I proposed the API that I think is more natural and IMO more desirable long-term.
Obviously the naming choice is just bike shedding and both would work, but max_pin=1.2.3 seems off to me, whereas upper_bound='x.x' is fine because it's a more generic term, where users don't have to guess what a pin is or isn't.
wolfv marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it support min_pin and max_pin? It's unclear right now in this spec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it does. Both use the same pin specification as mentioned earlier.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it support min_pin and max_pin? It's unclear right now in this spec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They both are derived from the same "pin" function. So yeah, they both support the same inputs.
chenghlee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
wolfv marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add an example of how it can be used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, what operators are available and what is the syntax?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some examples!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah so it's more like a version matcher. When I read compare I thought we were comparing two packages together like "is numpy's version greater than python's", which didn't make much sense to me.
But this is more like asking "Does the version of Python match this version expression?". So in that sense the name could be better expressed as:
version(python, "<3.8")version_match(python, "<3.8")match_version(python, "<3.8")satisfy(python, "<3.8")
Also this assumes that python will be represented by a concrete version x.y.z and not a spec like x.y.* right? IOW, it's already resolved to a specific package record.
If that's the case, should we specify that this function can only return after the solve has completed?
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we explicitly say that this is a read-only variable? E.g., is providing "hash: deadbeef123455" in the variant config file explicitly prohibited?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we call it variant_hash?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's not really prohibited in the implementation right now. But we can say that it should be...
I don't have strong feelings about the name. We can change it to variant_hash if that's preferred!
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we provide an example here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the first time we introduce the | syntax as well. Can we do version_to_buildstring(python)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah using a function would also work. But we also expose the default minijinja filters (such as lower, replace, ...), that can be used like ${{ "mystring" | upper }}
or ${{ val | replace('foo', 'moo') }}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jaimergp I added some more text regarding jinja filters to the bottom of the CEP.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this be overloaded to a single function? With the 2nd param being optional.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, that can work. We can also make it a named argument, e.g.
env.get("FOO", default="baz")
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jaimergp I implemented this change: prefix-dev/rattler-build#917
Let me know what you think.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpick. false and False are both used in the spec. It's hard to know if it returns a string or a bool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added the word boolean true to make it clear that this is a bool value.
Uh oh!
There was an error while loading. Please reload this page.