-
Notifications
You must be signed in to change notification settings - Fork 30
CEP XXXX: Virtual packages #103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 20 commits
ebcfa8e
cda0a8c
6f2d60f
9a793b9
9b4b7a7
59462ed
cedde6a
df9d61d
72022f4
7c3b595
e6fcda5
40b4442
bc6d6a6
5bf10f1
7a2c7e5
cd06910
e520336
1b01182
2148654
eaeb222
9a95c6b
6a9f82e
d4b32f9
edeb09c
30a6030
6b6714e
83ea61e
be15de8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,171 @@ | ||||||||||||||
| # CEP ???? - Virtual packages | ||||||||||||||
|
|
||||||||||||||
| <table> | ||||||||||||||
| <tr><td> Title </td><td> Virtual packages </td> | ||||||||||||||
| <tr><td> Status </td><td> Draft </td></tr> | ||||||||||||||
| <tr><td> Author(s) </td><td> Jaime Rodríguez-Guerra <[email protected]></td></tr> | ||||||||||||||
| <tr><td> Created </td><td> Dec 17, 2024</td></tr> | ||||||||||||||
| <tr><td> Updated </td><td> Dec 17, 2024</td></tr> | ||||||||||||||
| <tr><td> Discussion </td><td> https://github.com/conda/ceps/pull/103 </td></tr> | ||||||||||||||
| <tr><td> Implementation </td><td> https://github.com/conda/conda/tree/24.11.1/conda/plugins/virtual_packages, https://github.com/mamba-org/mamba/blob/libmamba-2.0.5/libmamba/src/core/virtual_packages.cpp, https://github.com/conda/rattler/tree/rattler-v0.28.8/crates/rattler_virtual_packages/src </td></tr> | ||||||||||||||
| </table> | ||||||||||||||
|
|
||||||||||||||
| > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", | ||||||||||||||
| "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as | ||||||||||||||
| described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. | ||||||||||||||
|
|
||||||||||||||
| ## Abstract | ||||||||||||||
|
|
||||||||||||||
| This CEP standardizes which virtual packages MUST be offered by conda install tools. | ||||||||||||||
|
|
||||||||||||||
| ## Motivation | ||||||||||||||
|
|
||||||||||||||
| Virtual packages are used to expose details of the system configuration to a conda client. They are commonly used as dependencies in regular packages to constrain on which systems they can be installed. Some examples include: | ||||||||||||||
|
|
||||||||||||||
| * On Linux, the minimum GNU `libc` version that must be available in the system via the `__glibc` virtual package. | ||||||||||||||
| * The oldest macOS version compatible with the package via the `__osx` virtual package. | ||||||||||||||
| * Whether a `noarch` package should be constrained to a single operating system via the `__linux`, `__osx` or `__win` virtual packages (often with no version constraint). | ||||||||||||||
| * The minimum CPU microarchitecture level that the binaries require via the `__archspec` virtual package. | ||||||||||||||
| * The lowest CUDA version the GPU driver is compatible with via `__cuda`. | ||||||||||||||
|
|
||||||||||||||
| ## Specification | ||||||||||||||
|
|
||||||||||||||
| A virtual package is defined as a package record with three fields: name, version and build string. | ||||||||||||||
| The name MUST start with double underscore (`__`). The version and build string MUST follow the same semantics as in regular package records. More specifically, the version field MUST follow the version string specifications, regardless its origin (computed from a system property, overridden by the user or configuration, or provided by default by the tool). | ||||||||||||||
|
|
||||||||||||||
| Some general considerations: | ||||||||||||||
|
|
||||||||||||||
| - The version or build string of a virtual package MAY be overridden by the value of `CONDA_OVERRIDE_{NAME}` environment variable, with `{NAME}` being the uppercased name of the virtual package. Many exceptions apply so please observe the details in the section below. | ||||||||||||||
jaimergp marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||||||||||
| - The build string MAY be zero (`0`). Some exceptions apply. See below. | ||||||||||||||
| - When the tool used a fallback default value instead of a computed one, it SHOULD also inform the user of that choice and its possible override options (e.g. `CONDA_OVERRIDE_{NAME}` variables, CLI flags, configuration file, etc). | ||||||||||||||
|
|
||||||||||||||
| ### List of virtual packages | ||||||||||||||
|
|
||||||||||||||
| In alphabetical order, every conda client MUST support the following virtual packages: | ||||||||||||||
|
|
||||||||||||||
| - `__archspec` | ||||||||||||||
| - `__cuda` | ||||||||||||||
| - `__glibc` | ||||||||||||||
| - `__linux` | ||||||||||||||
| - `__osx` | ||||||||||||||
| - `__unix` | ||||||||||||||
| - `__win` | ||||||||||||||
|
|
||||||||||||||
| #### `__archspec` | ||||||||||||||
|
|
||||||||||||||
| This virtual package MUST be always present, with the version set to `1`. | ||||||||||||||
|
||||||||||||||
|
|
||||||||||||||
| The build string MUST reflect one of: | ||||||||||||||
|
|
||||||||||||||
| - If the target platform matches the native platform, the best fitting CPU microarchitecture in the [`archspec/archspec-json` database](https://github.com/archspec/archspec-json/blob/v0.2.5/cpu/microarchitectures.json). The reference CPU detection implementation is [`archspec.cpu.detect.host()`](https://github.com/archspec/archspec/blob/v0.2.5/archspec/cpu/detect.py#L338). | ||||||||||||||
|
|
||||||||||||||
| - The target platform architecture (second component of the platform string), mapped as: | ||||||||||||||
|
||||||||||||||
|
|
||||||||||||||
| | Target platform | Reported `archspec` build string | | ||||||||||||||
| | --------------- | -------------------------------- | | ||||||||||||||
| | `*-32` | `x86` | | ||||||||||||||
| | `*-64` | `x86_64` | | ||||||||||||||
| | `*-armv6l` | `armv6l` | | ||||||||||||||
| | `*-armv7l` | `armv7l` | | ||||||||||||||
| | `*-aarch64` | `aarch64` | | ||||||||||||||
| | `*-arm64` | `arm64` | | ||||||||||||||
| | `*-ppc64` | `ppc64` | | ||||||||||||||
| | `*-ppc64le` | `ppc64le` | | ||||||||||||||
| | `*-riscv64` | `riscv64` | | ||||||||||||||
| | `*-s390x` | `s390x` | | ||||||||||||||
| | `zos-z` | `0` | | ||||||||||||||
| | Any other value | `0` | | ||||||||||||||
|
|
||||||||||||||
| The build string MUST be overridable with the `CONDA_OVERRIDE_ARCHSPEC` environment variable, if set to a non-empty value. | ||||||||||||||
jakirkham marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
baszalmstra marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||||||||||
|
|
||||||||||||||
| #### `__cuda` | ||||||||||||||
|
|
||||||||||||||
| This virtual package MUST be present when the system exhibits GPU drivers compatible with the CUDA runtimes. When available, the version value MUST be set to the oldest CUDA version supported by the detected drivers (i.e. the formatted value of `libcuda.cuDriverGetVersion()`), constrained to the first two components (major and minor) and formatted as `{major}.{minor}`. The build string MUST be `0`. | ||||||||||||||
|
|
||||||||||||||
|
||||||||||||||
| This virtual package MUST be present when the system exhibits GPU drivers compatible with the CUDA runtimes. When available, the version value MUST be set to the oldest CUDA version supported by the detected drivers (i.e. the formatted value of `libcuda.cuDriverGetVersion()`), constrained to the first two components (major and minor) and formatted as `{major}.{minor}`. The build string MUST be `0`. | |
| This virtual package MUST be present when the system exhibits NVIDIA GPU drivers compatible with the CUDA runtimes. | |
| For systems without such support the virtual package MUST NOT be present. | |
| When available, the version value MUST be set to the latest CUDA version supported by the detected drivers (i.e. | |
| the formatted value of [`cuDriverGetVersion()`]( https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VERSION.html )), | |
| constrained to the first two components (major and minor) and formatted as `{major}.{minor}`. The build string MUST be `0`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks John. With 12+, how is the upper bound determined? It's not about major.minor anymore, right? e.g. how do I know which y to use in <=12.y? Or are we expected to use <13?.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Each new CUDA Toolkit has a minimum driver version requirement, but has no upper bound. For example, you can run programs compiled with CTK 1.0 using the latest driver. You cannot run programs compiled with CTK 13.0 on machine that is running a driver from CTK 1.0. The CUDA driver has a stable ABI.
Therefore, __cuda=8.0 means that the newest CTK you can use is 8.0, but you can also use 7.5, 7.0, 6.5, ...
I hope that's clear.
jjhelmus marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we provide some guidance here on how the default value should be selected? If we don't, I could see situations arising where two different conda install tools or two different versions of the same install tool provide different default values on the same system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
conda/condasets it to2.5(yes, I know).- Pixi chose 2.28.
- Mamba skips it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rattler also skips the virtual package if a libc implementation cannot be found.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rattler the library? When does the pixi default come into play?
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Linux kernel upstream only defines {major}.{minor}.{micro}; anything beyond that (including the {patch} component you wrote) is part of the distribution kernel's version string. I don't think conda tooling should expose those components since their semantics of those will vary from distribution to distributon; e.g., patch 42 on Fedora may differ from patch 42 on Ubuntu.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, this contradicts this comment in conda/conda. I'm happy to trim to three fields, but we'll need a source.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah ha! Since Linux 3.0, "[mainline] kernels with 3-part versions are -stable kernels"; however, the mainline 2.6 series had 4-part version numbers. The existing "3- or 4-part" comment in conda/conda comes from the fact that 2.6 is the upstream for the CentOS/RHEL 6 kernel, which was actively supported by Anaconda and conda-forge at the time I wrote that patch.
In light of that, I suggest replacing:
Its version value MUST be set to the Linux kernel version, constrained to the first two to four numeric components formatted as
{major}.{minor}.{micro}.{patch}.
with something like:
Its version value MUST be set to the upstream (AKA mainline) Linux kernel version, but it MUST exclude any and all distribution-specific components of the kernel version.
I think that captures the intent of my previous comment, which was to ensure:
- we don't allow conda packages to depend on [Linux] distribution-specific kernels via the
__linuxvirtual package; and - we don't have to deal with the various ways Linux distros version their kernels (which may or may not be compatible with how conda does version ordering)
(That said, my suggested language does allow for non-release/development/-rcN kernels, but I suspect that user base is relatively small and able to cope if something goes horribly wrong.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added 6a9f82e
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will return the SYSTEM_VERSION_COMPAT version if the Python interpreter running the command was built against the 10.15 SDK or earlier.
See https://eclecticlight.co/2020/08/13/macos-version-numbering-isnt-so-simple/ for details.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAICR, if you start the Python interpreter with SYSTEM_VERSION_COMPAT=0 it returns the 11.x based version, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, running setting the environment variable will return a >=11 version.
My main concern here the "MUST NOT" language around the SYSTEM_VERSION_COMPAT workaround. I agree with the idea but this is not the case for the current version of conda (see conda/conda#13832). The example an this issue still reports a 10.16 version for __osx. Changing this to "SHOULD" would be reasonable, especially given that there are many releases of tools/packages that will report the compatible version that already exist.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we could consider that a bug in conda and submit a fix for 25.1 (as we are doing for __win). I can survey public repodata for __osx usage in the wild if that helps inform this decision.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Uh oh!
There was an error while loading. Please reload this page.