Discovery: Repodata v3 Backwards Compatibility for Wheels

[danyeaw]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

Determine if repodata v3 format, proposed in the draft CEP, can be backwards compatible with existing clients. Can older clients parse v3 files by ignoring new fields for wheels (extras, conditional dependencies, flags) while still using basic package metadata?

Why is this needed?

We want to make use of the repodata v3 format for supporting extras and conditionals when extending the repodata to include wheels, so this discovery should be focused on that use case.

Repodata v3 adds extras, conditional dependencies (pywin32; if __win), and flags. Currently, conda blocks any repodata_version > 2 with a CondaUpgradeError, which means deploying v3 would break all existing clients immediately. Backwards compatibility would enable gradual adoption without breaking users.

What should happen?

Investigate whether current parsing can handle v3 gracefully. For MatchSpec parsing - conditional dependencies extend syntax with ; if <CONDITION> in the depends list. Can existing parsers handle this or will they fail? Also verify solver compatibility and test with sample v3 repodata files.

Additional Context

No response

[jaimergp]

I feel the community is going to go in the direction of versioned repodata filenames. With fallbacks like we do with .zst compression. So we would have:

• v1: repodata.json, repodata.json.bz2, ...
• v2: repodata.v2.json, repodata.v2.json.bz2, repodata.v2.json.zst
• v3: repodata.v3.json

Or similar.

Alternatives include ensuring backwards compatibility while adding new stuff, which increases the complexity; e.g. adding new depends_v2 fields, or versioned package records (packages.conda.v2, or individual records like python-3.6-h123456: {"record_version": 2}.).

Also, are we assessing here in this ticket how to backport v3 to v1 for backwards compatibility.

---

Anyway, given my involvement in the CEP review so far, I'd like to get this assigned.

[jezdez]

I would also prefer if we'd use versions embedded in the transfer layer, either via HTTP request/response headers, or in the artifacts transmitted, e.g. as version numbers in the endpoints

[jaimergp]

Conditional dependencies review

Overview

The CEP PR#111 proposes three new features for match specs and package records, that result in a new repodata version that is backwards incompatible with versions v1 and v2:

• Conditional dependencies: add ; if <condition> to a depends field matchspec.
• Optional dependency groups (extras): a new dict[str, list[matchspec]] field in package records that extend the base depends group with its contents. Can be requested via matchspecs with package[extras="group1,group2"]
• Feature flags: a new iteration on build string globs (e.g. *gpu*) to allow for finer feature selection. Requires a new field in package records, and can be selected via matchspecs with package[flags="..."]. Some extra syntax allows to specify fuzzy matching and failure modes when missing.

The analysis below ignores the fact that the CEP proposes a repodata version bump to 3, and that conda will error out if that version is found in the regular repodata.json.

Extras and flags

Extras and flags introduce new record fields (as defined in CEP PR #133's index.json). This is generally ok, since tools will ignore extra fields. However, the requires MatchSpec syntax extensions cause errors:

>>> from conda.models.match_spec import MatchSpec as M
>>> M("package[extras='yes']")
conda.exceptions.InvalidMatchSpec: Invalid spec 'package[extras='yes']': Invalid spec 'package[extras='yes']': Cannot match on field(s): {'extras'}
>>> M("package[flags='gpu']")
conda.exceptions.InvalidMatchSpec: Invalid spec 'package[flags='gpu']': Invalid spec 'package[flags='gpu']': Cannot match on field(s): {'flags'}
>>>
>>> from rattler import MatchSpec as M
>>> M("package[extras='yes']")
exceptions.InvalidMatchSpecException: invalid bracket key: extras
>>> M("package[flags='gpu']")
exceptions.InvalidMatchSpecException: invalid bracket key: flags

Conditional dependencies

Conditional dependencies present more problems:

The conditional dependencies are simply added to the depends list, which would parsed by the solver trying to solve the environment. They won't be ignored. Best case, it results in parsing errors but... they are parsed by conda without errors! The result is wrong, unsurprisingly.

This is because conda does understand if as a potential component (at some point it must have been considered as a future improvement), and ignores anything coming after. The problem is that the semicolon ; is taken as part of the package name.

>>> from conda.models.match_spec import MatchSpec as M
>>> M("package; if python>3")
MatchSpec("package;")
>>> M("package[version='2.*']; if python>3")
MatchSpec("package;=2")
>>> M("package ; if __unix")  # adding a space doesn't work either
conda.exceptions.InvalidMatchSpec: Invalid spec 'package ; if __unix': Invalid version ';': invalid character(s)
>>> # Rattler 0.12 also chokes
>>> from rattler import MatchSpec as M
>>> M("package ; if __unix")
exceptions.InvalidMatchSpecException: unable to parse version spec: ; if __unix
>>> M("package[version='2.*']; if python>3")
exceptions.InvalidMatchSpecException: 'package[version' is not a valid package name. Package names can only contain 0-9, a-z, A-Z, -, _, or .

Pedantic note: if is a valid conda version (parsed as [[0], [0, 'if']]), so it is problematic for ill-formed packages like package;-if-__unix.conda because package; if __unix is a valid matchspec that should match that.

Wrapping the condition between parentheses would work because conda ignores those (part of an older syntax for "optional" dependencies that was removed):

>>> from conda.models.match_spec import MatchSpec as M
>>> M("package[version='2'] (python>=3)")
MatchSpec("package==2")

That said, I think the cleanest approach would be to use a key-value pair and focus more on those moving forward: package[if='python>=3']. Easier to integrate in the parsing engine, and clearer errors in older versions.

Conclusions

Extras and flags could be added to repodata v2 without too many backwards compatibility issues, as long as the new matchspec extensions are not used in the depends field. In other words, records could "offer" the new fields, but they cannot depend on them.

Conditional dependencies cannot be used in repodata v2. The require new matchspec syntax in depends, which is parsed by virtually every conda implementation. The obvious workaround of adding the conditional dependencies in a new field (e.g. depends_v2) so it is ignored by older clients would result in solution assymmetries: an older client won't take the conditional dependencies into account and produce a different solution than a newer client that does use them.

References

References:

• CEP: CEP for the next evolution of Repodata (v2) and MatchSpec ceps#111. Reviewed on 2025-11-16 at CEP for the next evolution of Repodata (v2) and MatchSpec ceps#111 (review).
• Implementations:
  • Implementation for resolvo: feat!: add support for conditional dependencies prefix-dev/resolvo#136
  • Implementation in rattler for resolvo backend: feat: parse matchspec conditions and translate to resolvo rattler#1545 + refactor: make extras and conditionals runtime-configurable rattler#1816
  • Implementation in rattler for libsolv backend: feat: add condition dependency and extras support for libsolv rattler#1836
• Tracking issues upstream:
  • Update repodata and MatchSpec support for conditionals / flags / extras based on latest CEP draft rattler#1790
  • Support for repodata and MatchSpec for conditional / extra / flags mamba-org/mamba#4089
• Discovery tickets:
  • Discovery: conditional and extra dependency support in libmamba conda-planning#106
  • Prototype conda-rattler-solver for repodata v3 support conda-incubator/conda-pypi#160
  • Discovery: early testing of rattler-solver with repodata v3 conda-incubator/conda-pypi#168
  • Discovery: Repodata v3 Backwards Compatibility for Wheels #15380