CEP: Standardize Lock File Format

[travishathaway]

To facilitate cross tool compatibility (e.g. conda, mamba, and pixi), we should standardize a lock file format so that all tools are able to read and write the same lock files.

Currently, there are two options:

• conda-lock
• rattler-lock

The latter lock file type has seen more recent development, and made several improvements over what was initially developed by conda-lock; therefore, it would make a better choice to use as the model for standardization.

The CEP to be written shall include a precise description of how the lock file is to be formatted and include a reference link to the current implementation in rattler-lock.

[jaimergp]

xref #79

[travishathaway]

Also: #81

[h-vetinari]

Tangentially related: PEP 751, where a multi-year discussion is now winding down towards a possible conclusion.

[travishathaway]

@h-vetinari,

Thanks for linking to that. I will be carefully reading that proposal and learn more about the ongoing discussions.

[jaimergp]

Hm, I thought the Python lockfiles were not going to make it in the end? Source: https://bsky.app/profile/snarky.ca/post/3ldp5zvst4c2v

[h-vetinari]

It got trimmed down because people (still) couldn't agree on much of the semantics. But the discussion has lots of material on this subject.

[h-vetinari]

New iteration of PEP 751: https://discuss.python.org/t/pep-751-one-last-time/77293

[jaimergp]

Currently, there are two options:

• conda-lock
• rattler-lock

To be more specific, there are nine options 😬

• conda has @EXPLICIT lockfiles (CEP at [CEP 23] Text spec input files #79)
• conda-lock has two versions, with v2 marked as preliminary.
• rattler-lock has six revisions so far.

[jjhelmus]

An argument can be made that some versions of environment.yaml files (c.f. #81 ) are lock file, specifically if they include the version and build string of all packages. In certain cases, but not all, this information is sufficient to identify a package exactly in the specified channels. That said there are a lot of details that these files miss so they should not be used as a starting point for a lock file standard IMHO.

Perhaps an alternative here it to determine what a lock file should include.

[jaimergp]

An argument can be made that some versions of environment.yaml files (c.f. #81 ) are lock file, specifically if they include the version and build string of all packages

These would invoke the solver with fresh repodata, which may have been patched in the meantime and thus is now maybe unsolvable. Or maybe a package was removed / marked as broken. So I would say that even with a 100% pinned environment.yml, you have to be very lucky to have it behave as a lockfile.

My definition for a conda lockfile includes the following properties:

• No solver or repodata checks involved
• File includes all data necessary to:
  • download the necessary packages,
  • verify their integrity with a good enough checksum,
  • and build the transaction to create the environment from scratch (or update to match the given state in the lockfile)

[baszalmstra]

For what its worth, these were the design goals of rattler_lock: https://docs.rs/rattler_lock/latest/rattler_lock/#design-goals

[jaimergp]

New iteration of PEP 751: discuss.python.org/t/pep-751-one-last-time/77293

Brett has just submitted it for evaluation: https://discuss.python.org/t/pep-751-one-last-time/77293/135