external variants pulled in first

[beckermr]

Similar to openmpi, the mamba solver is pulling in the external variants first. We should deploy the same fix here, namely making sure the external packages have the same deps as the real ones to help the solver.

I have also made a virtual package https://github.com/regro/conda-forge-conda-plugins with the mpich version so that we can try that out.

[beckermr]

cc @jaimergp @minrk

[beckermr]

xref: conda-forge/openmpi-feedstock#153 (comment)

[minrk]

I've come back to this, and I think it may come down to feature minimization, because if there's even one feature in any dependency of the 'real' builds, 'external' will be preferred, no matter how old the build or version is.

weighing down 'external' with enough features may do the trick. For it to work, though, we would need to ensure that external builds have at least as many features as all dependencies combined (plus one?). We would also have to make sure we apply the features to all past external builds.

How would we go about requiring the plugin virtual package? I think it's a really cool idea.

[beckermr]

The plugin exists and works. However, it only works with conda, and not mamba or rattler. So we'd have to solve that issue first. The problem is that the plugin is written in python and mamba + rattler do not have support for it. I made a proposal to fix this as a cep by moving towards executables as plugins but it needs to be voted on and then implemented.

[beckermr]

The cep is here: conda/ceps#78

[beckermr]

Well we said "temporary fix" and here we are ~1.5 years later with no movement. Sigh.

The work on non-python plugins stalled over in conda-land, and I don't really see that picking back up due to the issues raised.

In the mean time, now a colleague of mine has hit a problem that is the side effect of PRs #96 and #105.

• The latest external builds pull in conda-forge's libfabric due to link external libfabric #105.
• The conda-forge libfrabric is not ABI compatible with the HPC system one, causing issues at runtime since the HPC MPI libs link to the conda-forge libfabric.

In general, we're now hitting a "host" of problems with host deps for MPI libraries (pun intended!).

If we plan to swap out the conda-forge MPI build with the HPC system one due to the fact that we know the two are ABI compatible, then we

• have to do the same for any libraries that both the system one and the conda-forge one link to at run time
• the dependent libraries need to have their own ABI compatibility maintained
• could maybe hit weirdness if libraries are bundled in one build but not in others

Thoughts @minrk @dalcinl @isuruf @jaimergp?

[beckermr]

cc @joezuntz who put together a test case: https://github.com/joezuntz/mpi-nersc-test

[beckermr]

It may also very well be the case that the HPC vendor in this case shipped a custom libfabric that breaks ABI compabibility. In that case, there is actually nothing for conda-forge to do IMHO.

[beckermr]

Going back to bundling everything would fix the issue, but only if no other conda-forge package in the environment links against the host deps of MPI. In that case, we'd likely still see issues, maybe not with MPI itself, but with these potential other conda-forge packages.

[minrk]

Can we go back to an empty external package with a larger feature penalty? I think as long as the feature penalty on external is greater than the sum of available features in dependencies, it should work and avoid the external default install problem. I'd guess 2 or 3 is probably plenty, and we can always turn it up.

This same thing comes up on blas, e.g. conda-forge/blas-feedstock#149 where the default openblas has its own openmp feature, leading to a tie with other implementations meant to be penalized at the blas level.

You're right that there's going to be a conflict in libfabric if other packages in the env link libfabric directly, but I expect that to be rare, if ever. numcosmo appears to be the only one right now that's not an mpi provider:

$ mamba repoquery whoneeds --platform linux-64 --remote libfabric1 | awk1 | uniq
Getting

conda-forge/linux-64
conda-forge/noarch
Name
─────────────────
libfabric
libfabric-bin
libfabric-devel
mpich
mvapich
numcosmo
openmpi

and a quick check with readelf -d of numcosmo's outputs, it doesn't actually link libfabric, so even that seems to be a mistake.

[beckermr]

Yes we can pump up the features and (add a build number offset for good measure). Also agreed that numcosmo having libfabric in host is a mistake. IDK how or why that happened.

For completeness here is the error message my colleagues get when things go wrong:

MPICH ERROR [Rank 0] [job id 45659351.0] [Tue Nov 25 16:14:45 2025] [nid006578] 
- Abort(1617295) (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI 
error, error stack:
MPIR_Init_thread(170).......: 
MPID_Init(515)..............: 
MPIDI_OFI_mpi_init_hook(580): 
open_fabric(1502)...........: OFI fi_getinfo() failed (ofi_init.c:1502:open_fabr
ic:No data available)

aborting job:
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(170).......: 
MPID_Init(515)..............: 
MPIDI_OFI_mpi_init_hook(580): 
open_fabric(1502)...........: OFI fi_getinfo() failed (ofi_init.c:1502:open_fabr
ic:No data available)
MPICH ERROR [Rank 0] [job id 45659351.0] [Tue Nov 25 16:14:45 2025] [nid006578] 
- Abort(1617295) (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI 
error, error stack:
MPIR_Init_thread(170).......: 
MPID_Init(515)..............: 
MPIDI_OFI_mpi_init_hook(580): 
open_fabric(1502)...........: OFI fi_getinfo() failed (ofi_init.c:1502:open_fabr
ic:No data available)

The host libfabric appears to be libfabric.so.1 => /opt/cray/libfabric/1.22.0/lib64/libfabric.so.1 which should be ABI compatible.

A further twist is that apparently this all worked a while back. I am still waiting on details for those environments. It could be that libfabric is a red herring here and there is actually another issue.

[beckermr]

More information trickling in from my excellent colleagues! Thank you!

From @joezuntz

The cray libfabric claims to be version 1.22.0 in the directory name, but the library file is libfabric.so.1.25.0.
The only version symbol in the library is fi_version@@FABRIC_1.0

@heather999 confirms that the working environments she has found so far are from 2023 and so do not have conda-forge's libfabric.