In this case, the Nvidia MPS daemon has to be used. Otherwise, the
different processes cannot access the GPU memory of the other processes
as needed for the CUDA-aware MPI implementation because the memory
belongs to different contexts. When using MPS, the different contexts
are merged under the hood.

In any case it is probably a good idea to run MPS.