CEP: SBOM for package contents

[xhochy]

While we strive to avoid vendored dependencies in conda packages, they still happen under certain circumstances, e.g.

• If a specific commit of a C/C++ Library is required
• For languages where vendoring/static linkage is the norm: Haskell, Rust, Go, JavaScript

In these situations, the conda package metadata does not sufficiently cover the actual package contents. This situation is often described as having Phantom Dependencies. While some languages (e.g. Go) already contain sufficient information in the binary itself, it is costly to extract them. Thus, we should precompute them during the build, where we have the full dependency information

The generated SBOMs should be part of the files installed into the environment so that SBOM generation tools can generate high-quality SBOMs from the environment itself without the need to parse repodata or have a specialized tool for conda.

Implementation Idea (for rattler)

I would really like if rattler-build had a flag to generate an SBOM from package contents. It could be as simple as running syft over the directory. But already having the metadata computed once from the binares would save a lot of time. E.g. for a typical go binary, it took me 8s to do binary inspection with syft. If an SBOM were already present, this would be a matter of milliseconds.

Once we have decided where we want to put the SBOM in the package, pixi could have a pixi sbom that extracts the SBOM from the packages and uses the pixi.lock information itself to generate an SBOM.

My initial idea:

• Compute SBOMs as part (or after) of the package build
• Store them as part of the package, but ensure it ends up in conda-meta/ on installation
• If someone wants to generate an SBOM of a conda environment, it should be sufficient to run the tool on top of the conda-meta folder (e.g. in the case of syft, using only the conda and the SBOM cataloguer).
• Only generate SBOMs for packages that vendor something, not for all.

[jaimergp]

Linking some resources in the context of the recently approved PEP 770, which might come handy:

• The PEP itself: https://peps.python.org/pep-0770/
• More content at https://github.com/psf/sboms-for-python-packages (don't miss the issue tracker discussions!)
• Support for PEP 770 (SBOMs) mesonbuild/meson-python#763
• Record repaired libraries' provenance into wheel with SBOM pypa/auditwheel#541
• etc

The main take-away, I think, is that some of the SBOMs can be generated automatically, but in some cases they might also need to be provided manually, so we'll need some fields in the (meta|recipe).yaml files, or a convention similar to menuinst's Menu/ directory (but I really prefer dedicated metadata, likely in the about section next to the license info).

[JeanChristopheMorinPerso]

While we strive to avoid vendored dependencies in conda packages, they still happen under certain circumstances, e.g.

• If a specific commit of a C/C++ Library is required
• For languages where vendoring/static linkage is the norm: Haskell, Rust, Go, JavaScript

I don't know if it matters, but there are other reasons too. Some are hard to detect, leading us to forget or miss them, like C/C++ header only libraries. They go unnoticed very easily. Or DSOs in sdists, which also go unnoticed very easily. Or libraries being silently statically linked instead of dynamically linked.

[danielnachun]

While we strive to avoid vendored dependencies in conda packages, they still happen under certain circumstances, e.g.

• If a specific commit of a C/C++ Library is required
• For languages where vendoring/static linkage is the norm: Haskell, Rust, Go, JavaScript

I don't know if it matters, but there are other reasons too. Some are hard to detect, leading us to forget or miss them, like C/C++ header only libraries. They go unnoticed very easily. Or DSOs in sdists, which also go unnoticed very easily. Or libraries being silently statically linked instead of dynamically linked.

These are all very good additional examples in the C/C++/Fortran ecosystem. Header-only libraries seem like they will be incredibly difficult to detect.

As for silent static linking, I wonder if we have a way to inspect binaries for evidence of static linking. It is trivial of course to detect dynamic linking, but I wonder if there is some way to at least get the symbols that were statically linked. Though I don't really know how to distinguish symbols from the package itself from symbols from static libraries.

Regardless I will add these various scenarios to our list of situations where we don't currently have a good SBOM solution so we're aware that those areas needs more work.

[danielnachun]

Below is a summary I originally posted on Zulip of SBOM support from syft for almost all of the language ecosystems that we are likely to encounter in conda-forge. Any language with a green check mark means syft should be able to generate the SBOM at build time with no additional information or extra tools. Please feel free to update as needed.

Modular dependencies (no bundling)

• Python ✅
• Perl ⛔
• Lua ✅
• Ruby ✅
• R ✅

Bundled dependencies

• JDK JAR ✅
• .NET folder ✅
• NodeJS folder ⛔
• Go binary ✅
• Rust binary ⛔ - we have cargo-auditable for this already
• Ocaml binary ⛔
• Haskell binary ⛔
• GraalVM binary ⛔
• Statically linked C/C++/Fortran binary - Probably not
• Statically linked Swift binary - Probably not, but we may be able to force dynamic linking one we have a tool chain.

[danielnachun]

Here are my proposed solutions for some of the languages which don't currently work with syft:

• Perl isn't too much of a concern because it is modular, so there are no phantom dependencies, but for environments we'll have rely on the Conda metadata rather than syft directly detecting the modules. I'll note that Add package cataloger for perl ecosystem anchore/syft#1906 has been opened asking for Perl support so hopefully that is added in the future
• Ocaml binary - it is suggested here to use opam switch export --full --freeze and I've confirmed this produces a good SBOM.
• Haskell binary - cabal-plan show generates a JSON that can be used as an SBOM. Some extra parsing will be needed to remove extraneous stuff but the necessary information is there.
• GraalVM binary - we can use syft on the JAR file that is created during the build process before native compilation. Some work is needed to figure which JAR file this actually is, but I believe it must always be a single JAR.
• NodeJS (noarch) - I've found https://github.com/CycloneDX/cdxgen generates a valid SBOM.

[danielnachun]

Here are the scenarios where we don't have a good solution:

• NodeJS (native) - I'm much more concerned about NodeJS packages that include bundled native binaries in them. I have not yet found a way to force them to even build these bundled binaries from source, and they are almost always statically linked. While cdxgen can detect the dependency as a NodeJS module, it does not know if that dependency has native binaries. Right now unfortunately the only way to explicitly find all of them is to let ratter-build/conda-build find all the binaries. I imagine this would then require manually generating SBOMs for each package that contains statically linked native code. This may be a good question to raise with the NodeJS community.
• Bazel - SBOMs aren't really supported right now: What is status and timeline of SBOM work? bazelbuild/bazel#22966. I don't have much hope of this getting better any time soon given that the linked issue explicitly says SBOM support has been "deprioritized".
• gn (for Chromium/v8) - https://groups.google.com/a/chromium.org/g/chromium-dev/c/iAhQVHP5Uag provides some suggestion on how to do this for Chromium (and by extension qt-webengine), which may also apply to v8. This may also be a problem for other Webkit derivatives as well but I'm not sure.
• vendored/statically linked C/C++/Objective-C/D/Fortran libraries - This seems to be most common in CMake projects, though it can happen as well for autotools or Meson. There are several scenarios where this can happen:
  • A vendored library is used as a VCS submodule or is downloaded during the build process - sometimes this is a tagged release, while in other cases it may just a VCS revision. It may be the original upstream or a forked/patched version.
  • Vendored headerless libraries may be even harder to detect because they don't even produce separate object files, they just get inserted into an existing one
  • "Silent" statically linked conda-forge or host OS libraries - while we have a policy that generally tries to delete static libraries from our own packages, some packages still have them. And we cannot get rid of static libraries provided by the host OS, especially on macOS.
  • While R, Ruby, Lua and Perl don't produce binary wheels like Python does, it's rare but not unheard of for them to statically link and even vendor libraries (both header only and compiled) when they have a native code component. Native Python packages that are built from source have also been observed to statically linked and vendor libraries as well.

[danielnachun]

Given the lack of working toolchains for Haskell, Ocaml, GraalVM and Swift, I think the only actionable item right now is NodeJS, where we may want to get cdxgen added to conda-forge and then start requiring it to be run similar to pnpm-licenses. This is of course contingent on us standardizing where the SBOMs will be stored and how to declare them in recipes.

We will also eventually need to have a discussion about how strict we will be about new recipes for which SBOM generation is difficult or impossible with current tools - the main types of recipe submissions that would be affected at the present moment are native NodeJS packages, which can easily be detected, and any packages that vendor/statically link C/C++/Objective-C/D/Fortran libraries, which can be harder to detect. Though I think on macOS and Linux (and maybe Windows?) we can at least distinguish between dynamic and statically linked binaries, so the detection of statically linked binaries should at least raise suspicion that phantom dependencies may be present. That may even be detectable by some type of linting.

[JeanChristopheMorinPerso]

For ELF files created when compiling with FCC, I'm aware of https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-binaries# which is an attempt at tracking these kind of things. But I'm not aware of any large scale ecosystem actually making use of that.

[danielnachun]

For ELF files created when compiling with FCC, I'm aware of https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-binaries# which is an attempt at tracking these kind of things. But I'm not aware of any large scale ecosystem actually making use of that.

Thanks for sharing this, it was a very interesting read! As you noted, there isn't yet a consistent effort to use this broadly - I couldn't find an equivalent to the annobin plugin for LLVM or MSVC, it's unclear what the equivalent of ELF notes would be for MachO and PE binaries, and I don't think that syft or other SBOM tools know yet how to extract the information from these Watermark-annotated files.

However this type of effort does illustrate the point that the preprocessor and linker do in fact know a lot more about what was used to build the binary, but that information does not usually get annotated into the final binary. I would suspect that reason historically this didn't happen was because the Unix tradition was to compile everything from source locally, and Windows was rather infamously lax with regards to the security of pre-compiled binaries.

Now that both the US and EU have directives and regulations requiring proper SBOMs for software packages, I think this may help to motivate ecosystems/toolchains that haven't been very consistent about these types of annotations to start taking them more seriously. Specifically in the case of LLVM and MSVC when we have the opportunity to discuss SBOMs with those communities we can point them towards the GCC Annobin efforts as a starting point for what will be needed for better SBOM support.

[jaimergp]

I couldn't find an equivalent to the annobin plugin for LLVM or MSVC

According to the their docs, they also have a Clang and LLVM plugin (but for ELF only).

• Some more refs for annobin (these are in the blog post, but posting them here for visibility):
  • The watermark specification: https://fedoraproject.org/wiki/Toolchain/Watermark
  • https://sourceware.org/git/annobin.git
  • https://sourceware.org/annobin/

---

About macOS, the Mach-O file format doesn't seem to have a direct equivalent to ELF Notes, but LIEF has some docs about Mach-O modification that can be useful.

On Windows, I guess the way to go is the .rsrc section, paired with rc.exe and a resource configuration file? Our beloved LIEF also allows that.

[danielnachun]

@jaimergp That is great information on how we could hypothetically embed dependency information into Mach-O and PE binaries. It actually seem then that the harder part may be figuring out which headers and static libraries are from the package we are actually trying to build and which ones are vendored or being taken from another Conda package.

Since Go/Rust/Haskell/Ocaml/GraalVM by design cannot normally reuse previously compiled dependencies, we can be quite confident that whatever dependencies they declare in the project are in fact the only ones they really use. Even when they vendor and statically link C/C++/Fortran libraries, those libraries are brought in by a wrapper that is an explicit dependency whose version and licensing information we know, with the option in some cases to devendor by dynamically linking instead. So while we may need some language specific tools or commands to generate dependency metadata, the information is already there.

That is not true for vendoring in C/C++/Fortran packages. Sometimes the vendoring happens through git submodules, while in other cases the dependencies are downloaded at build time. Cmake and meson even provide explicit methods for downloading the dependencies. While could scan the source tree for those when a recipe is being submitted, we would find many false positives because in some cases those vendored dependencies are only used for building test binaries that aren't actually shipped to the user. And there are of course cases where the build steps includes a shell script that just downloads the dependencies outside the actual build system

I think we can get a lot more control over vendoring in C/C++/Fortran by blocking network access after the source is downloaded, and carefully checking cases of packages use git submodules or downloading an archive which is not a tagged release as these can be signs that vendoring is likely to be present. That won't address cases where the vendored dependencies are just copied into the source tree, but it should catch any instance where the vendored dependencies don't actually live in the source tree itself.

There isn't going to be a single solution to this problem but I think if we can come up strategies to make it easier for us to "catch" instances of vendoring it will reduce the prevalence of phantom dependencies in the C/C++/Fortran ecosystem.

[jezdez]

@xhochy Minor point about your initial proposal: rattler-build is not a conda community project, so your wishes about implementation details should be kept outside this proposal, given that CEPs are a function of the conda governance policy. In other words, CEPs can't dictate implementation in non-conda projects (imagine using CEPs to dictate implementation details in Anaconda projects ;).

[isuruf]

@jezdez, isn't that contradictory to the comment at #104 (comment)
Specifically,

By the way, I would hope that projects like mamba or rattler/pixi would also propose CEPs, for their respective projects if it helps their projects.

[jezdez]

@isuruf True, at least for pixi! Rattler is a conda org project.