RuntimeError: Cannot re-initialize CUDA in forked subprocess.

[riyajatar37003]

i am using transformer model to generate embeddings inside a function and that function is apply on each row of dataframe using parallel_apply which throwing belwo error

RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/app/snow.atg_arch_only.home/users/ariyaz/AI_Search/ais_ml_embedding_eval/ais_ml/tevatron/hn_mine_/negative-mine-v2.py", line 93, in
dataframe["output"] = dataframe.parallel_apply(lambda row: retriever(row,index,model,
File "/tmp/.local/lib/python3.10/site-packages/pandarallel/core.py", line 333, in closure
results_promise.get()
File "/opt/conda/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method