nvcc application building error with CUDA 11.5.0

[timohyva]

Hi, ascent devs, Happy New Year!

I used build_ascent_cuda.sh built ascent v0.9.3 with cuda/11.5.0 on a cluster, which has zen2 EPYC. here is what in my build script:

env  enable_mpi=ON enable_fortran=OFF raja_enable_vectorization=OFF enable_tests=OFF ./build_ascent_cuda.sh

Building was successful. The problem appears when Makefile linker links all the object files. I put

LDFLAGS  += $(ASCENT_LINK_RPATH)
LDLIBS   += $(ASCENT_MPI_CUDA_LIB_FLAGS)

In the linking stage, this error pops:
nvcc fatal : Unknown option '-Wl,-rpath,/my/project/path/ascent/scripts/build_ascent/install/ascent-develop/lib'

I checked the ascent_config.mk and identified the -Wl,-rpath, $(ASCENT_DIR)/lib, $(CONDUIT_LINK_RPATH) of ASCENT_LINK_RPATH. Why nvcc complains it is unknown? I tried to ass -Xcomplier in front of -Wl, also doesn't help.

I guess there is something I don't understand. Could your share some light to me?

[timohyva]

After searching, I get some clues about this issue.

-Wl, -rpath is gccoption for passing run time dynamic lib path. nvcc doesn't have this option at all.

I have a Makefile system, in which nvcc is linker. Then it certainly doesn't know -Wl, -rpath, and -Xcompiler surely can't help. This is saying Ascent running time libs are actually linked by GNU ld, or other compiler/linker which can handle this option.

What I need to do seemingly is to modify my Makefile system's linker, right? Suggestions are welcomed.

[cyrush]

@timohyva nvcc is always leveraging another compiler + linker, maybe -XLinker can help?

[timohyva]

I solved this problem, but I still want hear you guys' thoughts and experiences with this type of things, also record the solution for others.

I used ascent 0.9.3, changed conduit to 0.9.2, mfem to v4.7 from v4.6. The building is done by script based on build-ascent_cuda.sh, and it was successful.

Then problem here comes in, my project is using Makefile and I set LDFLAGS and LDLIBS as mentioned above. I tracked out all the g++ flags with -Wl, -rpath come from variables of *.mk makefile of all components. They are simply defined as

XXX_RPATH = -Wl, -rpath, $(xxxxx)

and directly passed to ASCENT_LINK_RPATH and ASCENT_MPI_CUDA_LIB_FLAGS, as a result, nvcc doesn't know it should call gcc or other compiler to pass to linker.

My solution is add -Xcompiler before almost every -Wl, -rpath, xxxx which is used. This changed *.mk files in install/xxx/share and guarantee to pass every used linker flags to gnu ld.

Because makefiles are changed in the install pathes, the v0.9.3 branch is in fact changed, and seems I have to keep them as new developments, which Ascent upstream may not want them appear in PR. However, I think these changes are still needed for me in even develop branch of Ascent.

I observed there won't be such problem if one use CMake other than Makefile. I don't have much experience of using CMake with hybrid building, so I am curious how CMake fix this.

[timohyva]

@timohyva nvcc is always leveraging another compiler + linker, maybe -XLinker can help?

I finally solved it by putting -Xcompiler in front of nearly every -Wl. Does -XLinker offers simpler solution?

[cyrush]

@timohyva I think -Xcompiler and -Xlinker are effectively the same if the flags are linker flags.
Separating out the RPATH flags is a good approach to solve this. We can look at improving our config.mk files, we may have some challenges with flags passed from tpls.

[timohyva]

I think I can summarize this host compiler invoking issue after years of usage.

CMake generated RPATHs in *.mk files among different dependencies contain minor typos such missing quotes, missing parent pathes etc. I've encounter these typoes in conduit and mfem and I always need to eyeball them and manually correct them. This is matter for Makefile managed project.

So, beside to add -Xcompiler to every places host compiler is needed, one also need put eye on checking typos from broken path.