Hang on loading meshing on INL HPC with multiple nodes

[GiudGiud]

Describe the bug
The simulation of the PB-FHR reflector using Nek, through Cardinal, are hanging on INL HPC when reading the mesh. This is not the only failing input but it's the one I'm working on. David Reger (@regerdavid) might have more.

pbfhr/mark1/reflector.nek: Test spec: /scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/hpc_tests:2
pbfhr/mark1/reflector.nek: Working directory: /scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/cht
pbfhr/mark1/reflector.nek: Running command: ssh bitterroot2.hpc.inl.gov 'cd /scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/cht; FILE="/scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/cht/nek.hpc_submit"; for i in {1..40}; do if [ -e "$FILE" ]; then sbatch $FILE; exit $?; else sleep 0.25; fi; done; exit 1'
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: ################################################################################
pbfhr/mark1/reflector.nek: Begin runner_run output
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: Displaying last 100 lines due to -x/--sep-files
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: <<<   polynomialOrder = 3
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< [VELOCITY]
pbfhr/mark1/reflector.nek: <<<   viscosity = -100.0
pbfhr/mark1/reflector.nek: <<<   density = 1.0
pbfhr/mark1/reflector.nek: <<<   residualTol = 1.0e-8
pbfhr/mark1/reflector.nek: <<<   boundaryTypeMap = W, W, symy, W, v, o, W, W
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< [PRESSURE]
pbfhr/mark1/reflector.nek: <<<   residualTol = 1.0e-8
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< [TEMPERATURE]
pbfhr/mark1/reflector.nek: <<<   conductivity = -1500.5
pbfhr/mark1/reflector.nek: <<<   rhoCp = 1.0
pbfhr/mark1/reflector.nek: <<<   residualTol = 1.0e-8
pbfhr/mark1/reflector.nek: <<<   boundaryTypeMap = f, f, I, I, t, o, f, I
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: <<< #include "udf.hpp"
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void UDF_LoadKernels(occa::properties & kernelInfo)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   // Compute the inlet velocity boundary condition and propagate it to a kernel variable
pbfhr/mark1/reflector.nek: <<<   kernelInfo["defines/Vz"] = 1.0;
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<<   // Compute the inlet temperature and propagate it to a kernel variable
pbfhr/mark1/reflector.nek: <<<   kernelInfo["defines/inlet_T"] = 0.0;
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void UDF_Setup(nrs_t *nrs)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   mesh_t * mesh = nrs->cds->mesh[0];
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<<   // loop over all the GLL points and assign directly to the solution arrays by
pbfhr/mark1/reflector.nek: <<<   // indexing according to the field offset necessary to hold the data for each
pbfhr/mark1/reflector.nek: <<<   // solution component
pbfhr/mark1/reflector.nek: <<<   int n_gll_points = mesh->Np * mesh->Nelements;
pbfhr/mark1/reflector.nek: <<<   for (int n = 0; n < n_gll_points; ++n)
pbfhr/mark1/reflector.nek: <<<   {
pbfhr/mark1/reflector.nek: <<<     nrs->U[n + 0 * nrs->fieldOffset] = 0.0; // x-velocity
pbfhr/mark1/reflector.nek: <<<     nrs->U[n + 1 * nrs->fieldOffset] = 0.0; // y-velocity
pbfhr/mark1/reflector.nek: <<<     nrs->U[n + 2 * nrs->fieldOffset] = 1.0; // z-velocity
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<<     nrs->P[n] = 0.0; // pressure
pbfhr/mark1/reflector.nek: <<<   }
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: <<< void velocityDirichletConditions(bcData * bc)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   if (bc->id == 5)
pbfhr/mark1/reflector.nek: <<<   {
pbfhr/mark1/reflector.nek: <<<     bc->u = 0.0; // x-velocity
pbfhr/mark1/reflector.nek: <<<     bc->v = 0.0; // y-velocity
pbfhr/mark1/reflector.nek: <<<     bc->w = Vz;  // z-velocity
pbfhr/mark1/reflector.nek: <<<   }
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void scalarDirichletConditions(bcData * bc)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   if (bc->id == 5)
pbfhr/mark1/reflector.nek: <<<     bc->s = inlet_T;
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void scalarNeumannConditions(bcData * bc)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   if (bc->id == 1 || bc->id == 2 | bc->id == 7)
pbfhr/mark1/reflector.nek: <<<     bc->flux = bc->usrwrk[bc->idM];
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< #ifdef __okl__
pbfhr/mark1/reflector.nek: <<< void meshVelocityDirichletConditions(bcData *bc){}
pbfhr/mark1/reflector.nek: <<< void velocityNeumannConditions(bcData *bc){}
pbfhr/mark1/reflector.nek: <<< void pressureDirichletConditions(bcData *bc){}
pbfhr/mark1/reflector.nek: <<< #endif
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: loading udf kernels ... 
pbfhr/mark1/reflector.nek: done (0.669249s)
pbfhr/mark1/reflector.nek: benchmarking hot kernels ...
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 2.21 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 3.04 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 1.34 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 1.20 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 1.34 across ranks!
pbfhr/mark1/reflector.nek: JIT compiling kernels (this may take awhile if they are not in cache) ...
pbfhr/mark1/reflector.nek: done (28.5335s)
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: wdotp nFields=01 min/avg/max: 7.609e-05s 8.363e-05s 1.445e-04s  (avg local: 6.484e-05s / 3.264e+01GB/s)
pbfhr/mark1/reflector.nek: wdotp nFields=03 min/avg/max: 6.155e-04s 6.717e-04s 1.063e-03s  (avg local: 5.748e-04s / 8.592e+00GB/s)
pbfhr/mark1/reflector.nek: Ax: N=3 wordSize=64 GDOF/s=0.021336 GB/s=4.45052 GFLOPS/s=3.48962 constCoeff=0 poisson=0 kernelVer=0
pbfhr/mark1/reflector.nek: Ax: N=3 wordSize=64 GDOF/s=0.030309 GB/s=4.59799 GFLOPS/s=4.59799 constCoeff=1 poisson=1 kernelVer=0
pbfhr/mark1/reflector.nek: Ax: N=3 wordSize=32 GDOF/s=0.138877 GB/s=10.5341 GFLOPS/s=21.0681 constCoeff=1 poisson=1 kernelVer=0
pbfhr/mark1/reflector.nek: fdm: N=5 wordSize=32 GDOF/s=0.0923908 GB/s=2.23512 GFLOPS/s=11.6546 kernelVer=0
pbfhr/mark1/reflector.nek: Ax: Ndim=3 N=3 wordSize=64 GDOF/s=0.0353469 GB/s=3.35141 GFLOPS/s=5.78118 constCoeff=0 poisson=0 kernelVer=0
pbfhr/mark1/reflector.nek: initializing linAlg ...
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek:  reading fluid.re2                                                                                                                           
pbfhr/mark1/reflector.nek:  hdr:#v003   264597  3   264597 this is the hdr                                      
pbfhr/mark1/reflector.nek: slurmstepd: error: *** JOB 2693376 ON br277 CANCELLED AT 2025-07-06T00:21:59 DUE TO TIME LIMIT ***
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: ################################################################################
pbfhr/mark1/reflector.nek: End runner_run output
pbfhr/mark1/reflector.nek: 
[3617s ]  TIMEOUT pbfhr/mark1/reflector.nek FAILED (SLURM JOB TIMEOUT) [min_cpus=192,job=2693376]

This happens with this version of OpenMPI (other packages for completeness), which is the one used by MOOSE for all applications

Installed packages
Python (/opt/miniforge3/envs/moose/bin/python3): Python 3.13.5
git (/usr/bin/git): git version 2.43.5
Conda (/opt/miniforge3/condabin/conda): conda 25.5.1
Conda prefix: /opt/miniforge3/envs/moose
C compiler (/opt/openmpi/bin/mpicc): gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
C++ compiler (/opt/openmpi/bin/mpicxx): g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
F77 compiler (/opt/openmpi/bin/mpif77): GNU Fortran (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
F90 compiler (/opt/openmpi/bin/mpif90): GNU Fortran (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
MPI (/opt/openmpi/bin/mpiexec): Open MPI (5.0.7)
PETSc (PETSC_DIR=/opt/petsc): v3.23.0-9-g95934b0d393
libMesh (LIBMESH_DIR=/opt/libmesh): f0a69d16b43414eba6d1c1bfe8482ccd6bda220b
MFEM (MFEM_DIR=/opt/mfem): 4.8.1

It can run successfully with these other modules, but we don't want to have a Cardinal-specific openmpi version, too costly maintenance-wise.

module load gcc/10.5.0-gcc-8.5.0-to6d
module load cmake
module load **openmpi/4.1.5_ucx1.14.1**

To Reproduce
Run this case from the NRIC virtual test bed on one of INL's cluster (bitterroot for example)
https://github.com/idaholab/virtual_test_bed/tree/devel/pbfhr/mark1/reflector
Ping me for access to HPC if needed

Expected behavior
Load the mesh, run the case in ~20 minutes with 240 CPUs

Desktop (please complete the following information):

• OS: Rocky Linux 8.10 (Green Obsidian)
• Version: The current NekRS version in Cardinal
  https://github.com/neams-th-coe/nekRS/tree/48e408eb9a1f4de674efd243a982c84300c792e4

Additional context
@loganharbour @aprilnovak for awareness

Needed for VTB HPC testing. You can use the VTB charge number with @eshemon 's permission at ANL

[yslan]

Have you tried other Modular Component Architecture?

On my laptop (and some other ubuntu 2204 servers and a mac), the same location triggers a MPI_Win_Create error but I can bypass it via either ucx or rdma.

 nrs->init ...
 call nek setup
 reading channel.re2
 hdr:#v002     1024  3      256 this is the hdr
[mcswl082:00000] *** An error occurred in MPI_Win_create
[mcswl082:00000] *** reported by process [4037804033,0]
[mcswl082:00000] *** on communicator MPI COMM 7 DUP FROM 5
[mcswl082:00000] *** MPI_ERR_WIN: invalid window
[mcswl082:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[mcswl082:00000] ***    and MPI will try to terminate your MPI job as well)

$ ompi_info |grep -i osc
                 MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.2)

mpirun --mca osc ucx 

[GiudGiud]

I have not. I ll take a look. Not sure this is an option for automated testing either but I ll ask
Is NekRS using MPI3 RMA?

[yslan]

Not sure this is an option for automated testing either

You simply add it into the submit script. Or, you can also set it via an environment variable e.g. OMPI_MCA_osc=ucx

Is NekRS using MPI3 RMA?

Our re2 reader, set_proc_map in Nek5000, uses MPI_WIN_Create. So, yes.

[loganharbour]

No luck with ucx either; littered with:

[wr12n39:585018] ../../../../../ompi/mca/osc/ucx/osc_ucx_component.c:369  Error: OSC UCX component priority set inside component query failed

This is UCX 1.18.1

[yslan]

No luck with ucx either; littered with:

[wr12n39:585018] ../../../../../ompi/mca/osc/ucx/osc_ucx_component.c:369  Error: OSC UCX component priority set inside component query failed

This is UCX 1.18.1

Does your desired OMPI module support ucx? check ompi_info.
I thought you are trying to avoid openmpi/4.1.5_ucx1.14.1?

I'd also suggest testing a smaller example regarding MPI_Win_Create.

$ mpic++ t.cpp
$ mpirun --mca osc ucx -n 2 ./a.out
$ mpirun --mca osc rdma -n 2 ./a.out

where

$ cat t.cpp

#include <mpi.h>
#include <iostream>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    int myData = rank;     // Local buffer
    int winBuffer = -1;    // Exposed memory for remote access

    // Create a window exposing winBuffer to RMA
    MPI_Win win;
    MPI_Win_create(&winBuffer, sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &win);

    MPI_Win_fence(0, win);  // Start RMA epoch

    if (rank == 0 && size > 1) {
        int valueToPut = 42;
        // Put value into rank 1's window
        MPI_Put(&valueToPut, 1, MPI_INT, 1, 0, 1, MPI_INT, win);
    }

    MPI_Win_fence(0, win);  // End RMA epoch

    std::cout << "Rank " << rank << " sees winBuffer = " << winBuffer << std::endl;

    MPI_Win_free(&win);
    MPI_Finalize();
    return 0;
}

[loganharbour]

No issues with the simple example.

This is ucx 1.18.1, openmpi 5.0.7. With the following components:

$ ompi_info | grep osc
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.7)

[yslan]

Have you tried rdma?

[yslan]

If you can run the example, then it might not be a MPI_Win_Create issue
Might have to attach the process with gdb to get the exact location..

[loganharbour]

rdma is the case that hangs.

I'll work on getting a trace for both the rdma hang and a trace for the ucx error

[stgeke]

Can you please try to run the code with:

mpirun --mca osc p2pt ...

[loganharbour]

I'm confused - how is this not an issue exactly? The tool not running with a modern openmpi implementation is not an issue?

[loganharbour]

And, for the above, we don't have p2pt:

$ ompi_info | grep osc
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.7)

[stgeke]

The issue you're facing is related to your system configuration and not the code itself. Something is off with the MPI installation.
Sorry I didn't realize that p2pt is not available. Can you try to reproduce with an older MPI e.g. 4.x

[GiudGiud]

@meltawila can you please build Nek with a 4.x openMPI module on Bitterroot? (not the one that we know works)

Try the simple case in this reply
#646 (comment)

and if it works let's move on to the failing case. Should only take 5 nodes & 200 cores to run it
https://github.com/idaholab/virtual_test_bed/tree/devel/pbfhr/mark1/reflector

[meltawila]

I tested /virtual_test_bed/pbfhr/mark1/reflector/cht building Cardinal on Bitterroot with the modules: openmpi/4.1.7-gcc-13.3.0-xpfl, cmake/3.29.3.

with these modules I found:

ompi_info |grep -i osc
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.7)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v4.1.7)
                 MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.7)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.7)

so I was setting OMPI_MCA_osc as an environment variable and running with srun --mpi=pmix. Except for pt2pt, while running with 224 MPI tasks setting OMPI_MCA_osc to any of the other 3 options, I was getting an error here:

loading udf kernels ... 
done (0.00390249s)
benchmarking hot kernels ...
WARNING: kernel[0] timings differ by up to 1.23 across ranks!
WARNING: kernel[0] timings differ by up to 1.30 across ranks!
WARNING: kernel[0] timings differ by up to 1.12 across ranks!
WARNING: kernel[0] timings differ by up to 1.33 across ranks!
JIT compiling kernels (this may take awhile if they are not in cache) ...
done (4.4423s)

Ax: N=3 wordSize=64 GDOF/s=0.0567941 GB/s=11.8468 GFLOPS/s=9.289 constCoeff=0 poisson=0 kernelVer=0
Ax: N=3 wordSize=64 GDOF/s=0.0624764 GB/s=9.4779 GFLOPS/s=9.4779 constCoeff=1 poisson=1 kernelVer=0
Ax: N=3 wordSize=32 GDOF/s=0.112507 GB/s=8.53384 GFLOPS/s=17.0677 constCoeff=1 poisson=1 kernelVer=0
fdm: N=5 wordSize=32 GDOF/s=0.112721 GB/s=2.72695 GFLOPS/s=14.2191 kernelVer=0
Ax: Ndim=3 N=3 wordSize=64 GDOF/s=0.0760128 GB/s=7.20714 GFLOPS/s=12.4323 constCoeff=0 poisson=0 kernelVer=0
initializing linAlg ...

wdotp nFields=01 min/avg/max: 2.641e-03s 2.672e-03s 2.818e-03s  (avg local: 2.657e-03s / 1.530e+01GB/s)
wdotp nFields=03 min/avg/max: 7.749e-03s 7.800e-03s 7.953e-03s  (avg local: 7.734e-03s / 1.226e+01GB/s)
 reading fluid.re2                                                                                                                           
 hdr:#v003   264597  3   264597 this is the hdr                                      
EXIT: MPI_Win_allocate failed!          0
  
an error occured: dying ...

and I get the error:

*** The MPI_Type_free() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[br287:846276] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

what worked was setting OMPI_MCA_osc=pt2pt which continued the run normally

[GiudGiud]

Using pmi2 is not an option for us either, we need to be using openmpi5 as that's what we use for all the other apps

[yslan]

Question, does your srun use the same PMI/PMIx layer as your openmpi5 module?
For the case of my univ server, it doesn't, and they eventually tell us to use mpiexec wrapper directly

[GiudGiud]

EDIT: I am referring to openmpi5, not @meltawila 's prior test with openmpi4

We actually call mpiexec directly not srun.
Like this

<mpiexec ...> apptainer exec /path/to/image '</path/to/binary ...>'

Question, does your srun (mpiexec) use the same PMI/PMIx layer as your openmpi5 module?

How do I check that?

[GiudGiud]

Tried ompi_info but no luck

(base) [giudguil@bitterroot1 ~]$ ompi_info 
                 Package: Open MPI mooseadmin@bitterroot1 Distribution
                Open MPI: 5.0.5
  Open MPI repo revision: v5.0.5
   Open MPI release date: Jul 22, 2024
                 MPI API: 3.1.0
            Ident string: 5.0.5
                  Prefix: /apps/moose/local/openmpi/5.0.5-gcc13.3.0
 Configured architecture: x86_64-pc-linux-gnu
           Configured by: mooseadmin
           Configured on: Thu Jul 25 15:05:35 UTC 2024
          Configure host: bitterroot1
  Configure command line: '--prefix=/apps/moose/local/openmpi/5.0.5-gcc13.3.0'
                          '--enable-shared'
                Built by: mooseadmin
                Built on: Thu Jul 25 15:11:17 UTC 2024
              Built host: bitterroot1
              C bindings: yes
             Fort mpif.h: yes (all)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
                          limitations in the
                          /apps/local/gcc/13.2.0/bin/gfortran compiler and/or
                          Open MPI, does not support the following: array
                          subsections, direct passthru (where possible) to
                          underlying Open MPI's C functionality
  Fort mpi_f08 subarrays: no
           Java bindings: no
  Wrapper compiler rpath: runpath
              C compiler: /apps/local/gcc/13.2.0/bin/gcc
     C compiler absolute: /apps/local/gcc/13.2.0/bin/gcc
  C compiler family name: GNU
      C compiler version: 13.2.0
            C++ compiler: /apps/local/gcc/13.2.0/bin/g++
   C++ compiler absolute: /apps/local/gcc/13.2.0/bin/g++
           Fort compiler: /apps/local/gcc/13.2.0/bin/gfortran
       Fort compiler abs: /apps/local/gcc/13.2.0/bin/gfortran
         Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
      Fort optional args: yes
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: yes
      Fort BIND(C) (all): yes
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
       Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
            Fort PRIVATE: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
         Fort USE...ONLY: yes
           Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
         Fort MPI_SIZEOF: yes
             C profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, Event lib: yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: no
          MPI extensions: affinity, cuda, ftmpi, rocm, shortfloat
 Fault Tolerance support: yes
          FT MPI support: yes
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
         MCA accelerator: null (MCA v2.1.0, API v1.0.0, Component v5.0.5)
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v5.0.5)
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v5.0.5)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                 MCA btl: self (MCA v2.1.0, API v3.3.0, Component v5.0.5)
                 MCA btl: sm (MCA v2.1.0, API v3.3.0, Component v5.0.5)
                 MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.5)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v5.0.5)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.5)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.5)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v5.0.5)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v5.0.5)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v5.0.5)
               MCA mpool: hugepage (MCA v2.1.0, API v3.1.0, Component v5.0.5)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v5.0.5)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v5.0.5)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v5.0.5)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v5.0.5)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v5.0.5)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.0.5)
             MCA threads: pthreads (MCA v2.1.0, API v1.0.0, Component v5.0.5)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                 MCA bml: r2 (MCA v2.1.0, API v2.1.0, Component v5.0.5)
                MCA coll: adapt (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: basic (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: han (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: inter (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: libnbc (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: self (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: sync (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: tuned (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: ftagree (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA coll: monitoring (MCA v2.1.0, API v2.4.0, Component
                          v5.0.5)
                MCA coll: sm (MCA v2.1.0, API v2.4.0, Component v5.0.5)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v5.0.5)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v5.0.5)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.5)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.5)
               MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                MCA hook: comm_method (MCA v2.1.0, API v1.0.0, Component
                          v5.0.5)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                  MCA io: romio341 (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                 MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                  MCA op: avx (MCA v2.1.0, API v1.0.0, Component v5.0.5)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.5)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                          v5.0.5)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.5)
                MCA part: persist (MCA v2.1.0, API v4.0.0, Component v5.0.5)
                 MCA pml: cm (MCA v2.1.0, API v2.1.0, Component v5.0.5)
                 MCA pml: monitoring (MCA v2.1.0, API v2.1.0, Component
                          v5.0.5)
                 MCA pml: ob1 (MCA v2.1.0, API v2.1.0, Component v5.0.5)
                 MCA pml: v (MCA v2.1.0, API v2.1.0, Component v5.0.5)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.5)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v5.0.5)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v5.0.5)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v5.0.5)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v5.0.5)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v5.0.5)

[yslan]

I'm out of things to try...
Since you are using mpi wrapper directly, it should just use whatever you prescribe. And, rdma should just work...
Assuming env var (and mpi wrapper) between the login node and compute node, I don't see why it fails.

                MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.5)
                MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                         v5.0.5)
                MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.5)

[loganharbour]

This is an aside - but I don't think falling back to pt2pt is a proper solution. It was removed from openmpi starting in version 5 due to bugginess and a lack of support.

[stgeke]

For testing purposes, please try running Open MPI 5.x with --mca osc rdma --mca btl tcp

[GiudGiud]

I got this

cardinal-openmpi-x86_64:1a0275f [giudguil@wr5n32: ((HEAD detached at origin/devel)) ~/vtb_new_new/pbfhr/reflector/conduction ]
$ mpirun -n 112 --mca osc rdma --mca btl tcp  cardinal-opt -i nek.i 
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications.  This means that no Open MPI device has indicated
that it can be used to communicate between these processes.  This is
an error; Open MPI requires that all MPI processes be able to reach
each other.  This error can sometimes be the result of forgetting to
specify the "self" BTL.

  Process 1 ([[12934,1],17]) is on host: wr5n32
  Process 2 ([[12934,1],17]) is on host: wr5n32
  BTLs attempted: tcp

Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another.  This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used.  Your MPI job will now abort.

You may wish to try to narrow down the problem;

 * Check the output of ompi_info to see which BTL/MTL plugins are
   available.
 * Run your application with MPI_THREAD_SINGLE.
 * Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
   if using MTL-based communications) to see exactly which
   communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_mpi_instance_init failed
  --> Returned "Unreachable" (-12) instead of "Success" (0)
--------------------------------------------------------------------------
[wr5n32:00000] *** An error occurred in MPI_Init_thread
[wr5n32:00000] *** reported by process [847642625,17]
[wr5n32:00000] *** on a NULL communicator
[wr5n32:00000] *** Unknown error
[wr5n32:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[wr5n32:00000] ***    and MPI will try to terminate your MPI job as well)
--------------------------------------------------------------------------
prterun has exited due to process rank 17 with PID 0 on node wr5n32 calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).

running with

--mca osc rdma

it proceeds normally.
running with the second option, it fails the same as with both

mpirun -n 112 --mca btl tcp  cardinal-opt -i nek.i

[stgeke]

I guess you need to run --mca osc rdma --mca btl self,tcp

[GiudGiud]

I guess you need to run --mca osc rdma --mca btl self,tcp

Ok these options work on a single node (we don't have issues with single-node runs at this time).
I'll try on 2 nodes and get back to you

[GiudGiud]

@loganharbour seems to work on a regular 2 node run (crashes later unrelatedly)
Can we try this on civet?