Skip to content

CEP XXXX: Virtual packages #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jaimergp wants to merge 28 commits into
base: main
Choose a base branch
from
Open

CEP XXXX: Virtual packages #103

Changes from 4 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
ebcfa8e
Write CEP about virtual packages
jaimergp Dec 17, 2024
cda0a8c
Add some Windows methods
jaimergp Dec 17, 2024
6f2d60f
some more
jaimergp Dec 17, 2024
9a793b9
add details of glibc, linux and osx
jaimergp Dec 17, 2024
9b4b7a7
pre-commit
jaimergp Dec 17, 2024
59462ed
add discussion, implementation
jaimergp Dec 17, 2024
cedde6a
add general paragraph about CONDA_OVERRIDE_*
jaimergp Dec 17, 2024
df9d61d
Apply suggestions from code review
jaimergp Dec 17, 2024
72022f4
more details about archspec
jaimergp Dec 17, 2024
7c3b595
Merge branch 'virtual-packages' of github.com:jaimergp/ceps into virt…
jaimergp Dec 17, 2024
e6fcda5
link to existing issue
jaimergp Dec 18, 2024
40b4442
Apply suggestions from code review
jaimergp Dec 19, 2024
bc6d6a6
Move motivation further up
jaimergp Dec 19, 2024
5bf10f1
Rework `__glibc` specification
jaimergp Dec 19, 2024
7a2c7e5
Forbid `CONDA_OVERRIDE_UNIX`
jaimergp Dec 19, 2024
cd06910
format table
jaimergp Dec 19, 2024
e520336
Reword general considerations
jaimergp Dec 19, 2024
1b01182
Be stricter with override values and more lenient with fallback values
jaimergp Dec 19, 2024
2148654
reword __archspec
jaimergp Dec 19, 2024
eaeb222
discard generic CPU rules
jaimergp Dec 19, 2024
9a95c6b
clarify env var syntax
jaimergp Dec 21, 2024
6a9f82e
Apply Cheng's suggestions
jaimergp Feb 12, 2025
d4b32f9
Add riscv32 and sort table
jaimergp Feb 12, 2025
edeb09c
Merge branch 'main' of github.com:conda/ceps into virtual-packages
jaimergp Sep 26, 2025
30a6030
Rename
jaimergp Sep 26, 2025
6b6714e
pre-commit
jaimergp Sep 26, 2025
83ea61e
more pre-commit
jaimergp Sep 26, 2025
be15de8
Update references to existing CEPs or drafts
jaimergp Oct 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 127 additions & 0 deletions cep-????.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# CEP ???? - Virtual packages

<table>
<tr><td> Title </td><td> Virtual packages </td>
<tr><td> Status </td><td> Draft </td></tr>
<tr><td> Author(s) </td><td> Jaime Rodríguez-Guerra &lt;[email protected]&gt;</td></tr>
<tr><td> Created </td><td> Dec 17, 2024</td></tr>
<tr><td> Updated </td><td> Dec 17, 2024</td></tr>
<tr><td> Discussion </td><td> NA </td></tr>
<tr><td> Implementation </td><td> Several </td></tr>
</table>

> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT",
"RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as
described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here.

## Abstract

This CEP standardizes which virtual packages MUST be offered by conda solvers.

## Specification

A virtual package is defined as a package record with three fields: name, version and build string.
The name MUST start with double underscore. The version and build string MUST follow the same semantics as in regular package records.

### List of virtual packages

In alphabetical order, every conda client MUST support the following virtual packages:

- `__archspec`
- `__cuda`
- `__glibc`
- `__linux`
- `__osx`
- `__unix`
- `__win`

#### `__archspec`

This virtual package MUST be always present, with the version set to `1`. The build string SHOULD reflect the detected CPU microarchitecture. If it cannot be detected, the build string SHOULD be `0`.

The build string MUST be overridable with the `CONDA_OVERRIDE_ARCHSPEC` environment variable, if set to a non-empty value.

#### `__cuda`

This virtual package MUST be present when the system exhibits GPU drivers compatible with the CUDA runtimes. When available, the version value MUST be set to the oldest CUDA version supported by the detected drivers (i.e. the formatted value of `libcuda.cuDriverGetVersion()`), constrained to the first two components (major and minor) and formatted as `{major}.{minor}`. The build string MUST be `0`.

Copy link
Contributor

@baszalmstra baszalmstra Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cuDriverGetVersion returns the version of the installed driver, not the oldest CUDA version supported by the detected drivers. Due to backwards compatibility support newer drivers also support older versions.

Copy link
Contributor Author

@jaimergp jaimergp Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this returns the CUDA version as seen by the driver. The docs are not super clear. There's also cudaRuntimeGetVersionAt least this is what conda/conda uses.

@jakirkham could you help clarify this? Thanks!

Copy link
Contributor

@baszalmstra baszalmstra Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I think the driver CUDA version might not be the "oldest CUDA version supported by the detected driver". E.g. If my driver is 12.6 it might also support older CUDA versions.

Copy link
Member

@jakirkham jakirkham Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asking offline to see if others have thoughts here

That said, here are some rough thoughts. Starting with some background

When the CUDA Toolkit is typically shipped (like in the installer), it ships with a full suite of things including the driver library. However in Conda, we ship pretty much everything else except the driver library. So with Conda, the user is on the hook for ensuring the driver library is installed, which would happen when they install the driver

So this raises a question. How does Conda decide what version of the driver is compatible with a particular CUDA Toolkit?

This is what __cuda is solving. It is giving us the driver library's associated CUDA Toolkit version. IOW if that driver library was shipped in an installer, this is that installer's CUDA Toolkit version

So now when we try to install the CUDA Toolkit libraries, we can answer the question, will the driver be able to handle these?

Originally the answer was the CUDA Toolkit libraries we install could be no newer than the CUDA Toolkit the driver came from. We had checked this down to the major minor version (patch version was allowed to float). So if this driver check reported CUDA 10.1, we could only use CUDA Toolkit libraries from 10.1 or older. Eventually we relaxed this to major only for 11.2+ and 12

So really __cuda's version is the upper bound on the libraries we can use

Sorry for the long winded answer. Though hopefully the additional details provide context. Perhaps this can be incorporated into the text above somehow (after some discussion and Q&A)

Copy link
Member

@jakirkham jakirkham Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like this captures the idea above. Though please feel free to adjust further. Happy to look again after

Suggested change
This virtual package MUST be present when the system exhibits GPU drivers compatible with the CUDA runtimes. When available, the version value MUST be set to the oldest CUDA version supported by the detected drivers (i.e. the formatted value of `libcuda.cuDriverGetVersion()`), constrained to the first two components (major and minor) and formatted as `{major}.{minor}`. The build string MUST be `0`.
This virtual package MUST be present when the system exhibits NVIDIA GPU drivers compatible with the CUDA runtimes.
For systems without such support the virtual package MUST NOT be present.
When available, the version value MUST be set to the latest CUDA version supported by the detected drivers (i.e.
the formatted value of [`cuDriverGetVersion()`]( https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VERSION.html )),
constrained to the first two components (major and minor) and formatted as `{major}.{minor}`. The build string MUST be `0`.

Copy link
Contributor Author

@jaimergp jaimergp Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks John. With 12+, how is the upper bound determined? It's not about major.minor anymore, right? e.g. how do I know which y to use in <=12.y? Or are we expected to use <13?.

Copy link

@carterbox carterbox Nov 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each new CUDA Toolkit has a minimum driver version requirement, but has no upper bound. For example, you can run programs compiled with CTK 1.0 using the latest driver. You cannot run programs compiled with CTK 13.0 on machine that is running a driver from CTK 1.0. The CUDA driver has a stable ABI.

Therefore, __cuda=8.0 means that the newest CTK you can use is 8.0, but you can also use 7.5, 7.0, 6.5, ...

I hope that's clear.

The version MUST be overridable with the `CONDA_OVERRIDE_CUDA` environment variable, if set to a non-empty value.

#### `__glibc`

This virtual package MUST be present when the native platform is `linux-*`. Its version value MUST be set to the system `libc` version, constrained to the first two components (major and minor) formatted as `{major}.{minor}`. The build string MUST be `0`.

The `libc` version can be computed via:

- Python's `os.confstr("CS_GNU_LIBC_VERSION")`
- `getconf GNU_LIBC_VERSION`
- `ldd --version` (in GLIBC distros)
- System's `libc.so` (in MUSL distros, location not standardized)
Copy link
Member

@jakirkham jakirkham Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think MUSL should be its own thing. It is versioned differently. Plus users presumably want to distinguish GLIBC built packages from MUSL ones

Copy link

@h-vetinari h-vetinari Dec 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the stdlib infrastructure, we have the technical requirements in place to do something like:

c_stdlib:               # [linux]
  - sysroot             # [linux]
  - musl                # [linux]
c_stdlib_version:       # [linux]
  - 2.17                # [linux]
  - 1.2                 # [linux]
zip_keys:               # [linux]
  - - c_stdlib          # [linux]
    - c_stdlib_version  # [linux]

Whether we want to roll out a "conda-forge for musl" is of course a separate question.

For the purpose of this CEP, definitely __glibc needs to stay musl-free IMO. For future-proofing, we could add __musl, but no strong feelings.

Copy link
Contributor Author

@jaimergp jaimergp Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the GLIBC version in MUSL distros, not the MUSL version itself. Happy to adjust the wording though. See this block in the constructor SH templates for some more context.

Copy link

@h-vetinari h-vetinari Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having glibc in musl is certainly not the default, I actually didn't know this is supported - TIL. In that case I agree it makes sense to report the glibc that's found.

So my point was that we could also build musl-native libs, but I guess that's much less urgent if alpine-users can just install a glibc compat (assuming this works with our glibc-flavoured packages).

Copy link
Member

@jakirkham jakirkham Dec 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the original comment ( conda/constructor#850 (comment) ), think that user is referring to gcompat:

...a library which provides glibc-compatible APIs for use on musl libc systems.

While that can be useful, think we should not encode that in this CEP

It may make sense to have a CEP extending this for MUSL (perhaps including gcompat) and this could be one detail we include there, but would suggest we leave it out of this version and save it as future work

Copy link
Contributor Author

@jaimergp jaimergp Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that user is referring to gcompat:

I'm not sure the original report was about gcompat, but about this alpine-focused glibc version.

Copy link
Member

@jakirkham jakirkham Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The challenge is if we include this line (even as an example) future readers will look at this line and say "Hey that is how they said we should be supporting MUSL." Even if that is not what we meant. So taking this line out is safer than keep it IMO

The rest of the examples above seem fine

Copy link

@h-vetinari h-vetinari Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jakirkham: think that user is referring to gcompat:

Look at the alpine wiki link I posted:

[wiki] If you want to run glibc programs in Alpine Linux, there are a few ways of doing so.

gcompat is one of them, but there are others.

@jakirkham: future readers will look at this line and say "Hey that is how they said we should be supporting MUSL."

I don't think that's realistic; it's also easy to preempt - just say "the version of glibc, if present (musl-based distros like alpine only provide this for compatibility; their main C library is musl itself)"

Copy link
Member

@jakirkham jakirkham Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry Axel think we had posted around the same time. Agree there can be other ways people try to handle this need

To loop back to Jaime's original point on how to word this, what if we just say the following

Suggested change
- System's `libc.so` (in MUSL distros, location not standardized)
- System's GNU `libc.so` (see specific disto for location)

Note: Some distros put libc.so in different places. So this is worth stating in its own right

Copy link
Contributor Author

@jaimergp jaimergp Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworded these paragraphs a bit including the intent of the suggestions, let me know what you think!


The version MUST be overridable with the `CONDA_OVERRIDE_GLIBC` environment variable, if set to a non-empty value.

If the `libc` version could not be estimated (e.g. the tool is not running on Linux), the tool SHOULD provide a default value (e.g. `2.17`) and inform the user of that choice and its possible overrides; e.g. via `CONDA_OVERRIDE_GLIBC`, a CLI flag or a configuration file. The environment variable MUST be ignored when the target platform is not `linux~-*`.

#### `__linux`

This virtual package MUST be present when the target platform is `linux-*`. Its version value MUST be set to the Linux kernel version, constrained to two to four numeric components formatted as `{major}.{minor}.{micro}.{patch}`. If the version cannot be estimated (e.g. because the native platform is not Linux), the fallback value MUST be set to `0`. The build string MUST be `0`.

The Linux kernel version can be obtained via:

* Python's `platform.release()`
* `uname -r`
* `cat /proc/version`

The version MUST be overridable with the `CONDA_OVERRIDE_LINUX` environment variable, if set to a non-empty value that matches the regex `"\d+\.\d+(\.\d+)?(\.\d+)?"`. The environment variable MUST be ignored when the target platform is not `linux-*`.

#### `__osx`

This virtual package MUST be present when the target platform is `osx-*`. Its version value MUST be set to the first two numeric components of macOS version formatted as `{major}[.{minor}]`. If the version cannot be estimated (e.g. because the native platform is not macOS), the fallback value MUST be set to `0`. The build string MUST be `0`.

The macOS version can be ontained via:

- Python's `platform.mac_ver()[0]`
- `sw_vers -productVersion`

> If applicable, the `SYSTEM_VERSION_COMPAT` workaround MUST NOT be enabled; e.g. the version reported for Big Sur should be 11.x and not 10.16.

The version MUST be overridable with the `CONDA_OVERRIDE_OSX` environment variable. If this environment variable is set to the empty string `""`, then the `__osx` virtual package MUST NOT be present. The environment variable MUST be ignored when the target platform is not `osx-*`.

#### `__unix`

This virtual package MUST be present when the target platform is `linux-*`, `osx-*` or `freebsd-*`. The version and build string fields MUST be set to `0`.

The version or build string fields MUST NOT be overriden by the `CONDA_OVERRIDE_UNIX` environment variable. However, if this environment variable is set to a non-empty value, the `__unix` virtual package MUST be present if otherwise if would not have been. The environment variable MUST be ignored when the target platform is not `linux-*`, `osx-*` or `freebsd-*`.

#### `__win`

This virtual package MUST be present when the target platform is `win-*`. The version MUST be set to the first three numeric components of the Windows build version, formatted as `{major}.{minor}.{build}`. If the version cannot be estimated (e.g. because the target platform does not match the native platform), the fallback value MUST be set to `0`. The build string MUST be `0`.

The string `{major}.{minor}.{build}` can be obtained from:

- Python's `platform.win32_ver()`
- CMD's `ver`
- Powershell's `[System.Environment]::OSVersion.Version`, `(Get-CimInstance Win32_OperatingSystem).version`
- The command `wmic os get version`

The version MUST be overridable with the `CONDA_OVERRIDE_WIN` environment variable. If this environment variable is set to the empty string `""`, then the `__win` virtual package MUST NOT be present. The environment variable MUST be ignored when the target platform is not `win-*`.

## Motivation

Virtual packages are used to expose details of the system configuration to a conda client. They are commonly used as dependencies in regular packages to constrain on which systems they can be installed. Some examples include:

* On Linux, the minimum `libc` version that must be available in the system via the `__glibc` virtual package.
* The oldest macOS version compatible with the package via the `__osx` virtual package.
* Whether a `noarch` package should be constrained to a single operating system via the `__linux`, `__osx` or `__win` virtual packages (often with no version).
* The minimum CPU microarchitecture level that the binaries require via the `__archspec` virtual package.
* The lowest CUDA version the GPU driver is compatible with via `__cuda`.

## References

* [Virtual packages implementation in `conda/conda` 24.11.1](https://github.com/conda/conda/tree/24.11.1/conda/plugins/virtual_packages)
* [ENH: make `__win` version usable for package metadata (conda/conda#14443)](https://github.com/conda/conda/issues/14443)
* [Drop `CONDA_OVERRIDE_WIN` environment variable (mamba-org/mamba#2815)](https://github.com/mamba-org/mamba/pull/2815)

## Copyright

All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).