BUG: Compilation failure on HIP architecture AMD Instinct MI300A

[dutta-alankar]

Describe the issue:

I am trying to compile idefix to be run on our new AMD APU cluster. This code compiles and runs fine on an older cluster with Nvidia A100 GPUs.
First I try to run cmake which was successful.
CC=hipcc CXX=hipcc cmake -DIdefix_MPI=ON -DIdefix_DEBUG=OFF -DIdefix_HDF5=ON -DKokkos_ENABLE_HIP=ON -DKokkos_ARCH_AMD_GFX942_APU=ON -DKOKKOS_ENABLE_CXX17=ON -DCMAKE_HIP_ARCHITECTURES="gfx942" $IDEFIX_DIR
It gives the following output

-- The C compiler identification is Clang 20.0.0
-- The CXX compiler identification is Clang 20.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /mpcdf/soft/RHEL_9/packages/x86_64/rocm/7.0.2/bin/hipcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /mpcdf/soft/RHEL_9/packages/x86_64/rocm/7.0.2/bin/hipcc - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting default Kokkos CXX standard to 17
-- Kokkos version: 4.5.0
-- The project name is: Kokkos
-- Using internal gtest for testing
-- Configured git information in /ptmp/adutt/idefix-freya/test/HD/trml/generated/Kokkos_Version_Info.cpp
-- Compiler Version: 7.0.51831
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=gnu++17 for C++17 extensions as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::HIP
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
-- 
-- Architectures:
--  AMD_GFX942_APU
-- Performing Test Kokkos_ENABLE_IMPL_SKIP_NO_RTTI_FLAG
-- Performing Test Kokkos_ENABLE_IMPL_SKIP_NO_RTTI_FLAG - Failed
-- Found TPLLIBDL: /usr/include  
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS - Success
-- Using internal desul_atomics copy
-- Experimental mdspan support is enabled
-- Looking for C++ include experimental/mdspan
-- Looking for C++ include experimental/mdspan - not found
-- Looking for C++ include mdspan
-- Looking for C++ include mdspan - not found
-- Using internal mdspan directory /ptmp/adutt/idefix-freya/src/kokkos/tpls/mdspan/include
-- Kokkos Backends: SERIAL;HIP
-- Including problem-specific CMakeLists in '/ptmp/adutt/idefix-freya/test/HD/trml'
--     Adding problem-specific source file frameBoost.cpp
--     Adding problem-specific source file frameBoost.hpp
-- Found MPI_C: /viper/u2/system/soft/RHEL_9/packages/znver4/openmpi_gpu/gcc_14-14.1.0-rocm_7.0-7.0.2/5.0.8/lib/libmpi.so (found version "3.1") 
-- Found MPI_CXX: /viper/u2/system/soft/RHEL_9/packages/znver4/openmpi_gpu/gcc_14-14.1.0-rocm_7.0-7.0.2/5.0.8/lib/libmpi.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")  
-- Found HDF5: hdf5-shared (found version "2.0.0") found components: C 
-- XDMF (hdf5+xmf) dumps enabled
-- Idefix final configuration
--     MHD:  OFF
--     MPI:  ON
--     HDF5: ON
--     Python: OFF
--     Reconstruction: Linear
--     Precision: Double
--     Version: 2.2.01-6f7733f
--     Problem definitions: 'definitions.hpp'
-- Configuring done (10.7s)
-- Generating done (0.1s)
-- Build files have been written to: /ptmp/adutt/idefix-freya/test/HD/trml

Then on running make Kokkos gets compiled successfully and then compilation breaks as soon as it tries to compile the very first idefix source file. Following is the relevant part of the error message.

Error message:

[  2%] Built target kokkossimd
[ 36%] Built target kokkoscore
[ 39%] Built target kokkoscontainers
[ 41%] Building CXX object CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:283:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:29:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:929:19: error: static assertion failed due to requirement 'ViewMapping<Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::is_assignable': Incompatible View copy construction
  929 |     static_assert(Mapping::is_assignable,
      |                   ^~~~~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/output/scalarField.hpp:36:39: note: in instantiation of function template specialization 'Kokkos::View<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>::View<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   36 |       IdefixHostArray3D<real> arr3D = Kokkos::create_mirror(d3Darray);
      |                                       ^
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:283:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:29:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:202:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewMapping.hpp:2944:19: error: static assertion failed due to requirement 'is_assignable_space': View assignment must have compatible spaces
 2944 |     static_assert(is_assignable_space,
      |                   ^~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:931:14: note: in instantiation of member function 'Kokkos::Impl::ViewMapping<Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::assign' requested here
  931 |     Mapping::assign(m_map, rhs.m_map, rhs.m_track.m_tracker);
      |              ^
/ptmp/adutt/idefix-freya/src/output/scalarField.hpp:36:39: note: in instantiation of function template specialization 'Kokkos::View<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>::View<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   36 |       IdefixHostArray3D<real> arr3D = Kokkos::create_mirror(d3Darray);
      |                                       ^
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:283:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:29:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:944:19: error: static assertion failed due to requirement 'ViewMapping<Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::is_assignable': Incompatible View copy assignment
  944 |     static_assert(Mapping::is_assignable, "Incompatible View copy assignment");
      |                   ^~~~~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/global.hpp:41:14: note: in instantiation of function template specialization 'Kokkos::View<int *, Kokkos::LayoutRight, Kokkos::HostSpace>::operator=<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   41 |   outArrHost = Kokkos::create_mirror_view(outArr);
      |              ^
/ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:633:25: note: in instantiation of function template specialization 'idfx::ConvertVectorToIdefixArray<int>' requested here
  633 |   this->mapVars = idfx::ConvertVectorToIdefixArray(mapVars);
      |                         ^
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:283:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:29:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:202:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewMapping.hpp:2944:19: error: static assertion failed due to requirement 'is_assignable_space': View assignment must have compatible spaces
 2944 |     static_assert(is_assignable_space,
      |                   ^~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:945:14: note: in instantiation of member function 'Kokkos::Impl::ViewMapping<Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::assign' requested here
  945 |     Mapping::assign(m_map, rhs.m_map, rhs.m_track.m_tracker);
      |              ^
/ptmp/adutt/idefix-freya/src/global.hpp:41:14: note: in instantiation of function template specialization 'Kokkos::View<int *, Kokkos::LayoutRight, Kokkos::HostSpace>::operator=<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   41 |   outArrHost = Kokkos::create_mirror_view(outArr);
      |              ^
/ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:633:25: note: in instantiation of function template specialization 'idfx::ConvertVectorToIdefixArray<int>' requested here
  633 |   this->mapVars = idfx::ConvertVectorToIdefixArray(mapVars);
      |                         ^
4 errors generated when compiling for gfx942.
failed to execute:/mpcdf/soft/RHEL_9/packages/x86_64/rocm/7.0.2/lib/llvm/bin/clang++  --offload-arch=gfx942  -DKOKKOS_DEPENDENCE -DMHD=NO -DORDER=2 -DUSE_PROF_API=1 -DWITH_HDF5 -DWITH_MPI -D__HIP_PLATFORM_AMD__=1 -I/ptmp/adutt/idefix-freya/src/fluid/cooling -I/ptmp/adutt/idefix-freya/test/HD/trml -I/ptmp/adutt/idefix-freya/src/kokkos/core/src -I/ptmp/adutt/idefix-freya/src/dataBlock -I/ptmp/adutt/idefix-freya/src/dataBlock/planetarySystem -I/ptmp/adutt/idefix-freya/src/fluid -I/ptmp/adutt/idefix-freya/src/fluid/boundary -I/ptmp/adutt/idefix-freya/src/fluid/braginskii -I/ptmp/adutt/idefix-freya/src/fluid/constrainedTransport -I/ptmp/adutt/idefix-freya/src/fluid/eos -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver/HDsolvers -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver/MHDsolvers -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver/Dustsolvers -I/ptmp/adutt/idefix-freya/src/fluid/tracer -I/ptmp/adutt/idefix-freya/src/output -I/ptmp/adutt/idefix-freya/src/rkl -I/ptmp/adutt/idefix-freya/src/gravity -I/ptmp/adutt/idefix-freya/src/utils -I/ptmp/adutt/idefix-freya/src/utils/iterativesolver -I/ptmp/adutt/idefix-freya/src -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/core/src -I/ptmp/adutt/idefix-freya/src/kokkos/tpls/desul/include -I/ptmp/adutt/idefix-freya/src/kokkos/tpls/mdspan/include -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/containers/src -I/ptmp/adutt/idefix-freya/src/kokkos/containers/src -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/algorithms/src -I/ptmp/adutt/idefix-freya/src/kokkos/algorithms/src -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/simd/src -I/ptmp/adutt/idefix-freya/src/kokkos/simd/src -isystem /viper/u2/system/soft/RHEL_9/packages/znver4/openmpi_gpu/gcc_14-14.1.0-rocm_7.0-7.0.2/5.0.8/include -isystem /viper/u2/system/soft/RHEL_9/packages/znver4/openmpi_gpu/gcc_14-14.1.0-rocm_7.0-7.0.2/5.0.8/include/openmpi -isystem /mpcdf/soft/RHEL_9/packages/znver4/hdf5/gcc_14-14.1.0-openmpi_5.0-5.0.8/2.0.0/include -O3 -DNDEBUG -std=gnu++17 -fno-gpu-rdc -x hip -MD -MT CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o -MF CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o.d -o "CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o" -c /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp
make[2]: *** [CMakeFiles/idefix.dir/build.make:76: CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:643: CMakeFiles/idefix.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

runtime information:

Commit hash: ede2e1f5e30e4486bd1dfcf141b4a67126634093 with some additional changes.
Kokkos commit hash: 09e775bfc585840bb9ab1156cbd8d7d1c9e0cc6d

[glesur]

Hi @dutta-alankar , this appears to be a Kokkos internal error, possibly due to the fact that you're using a very rescent version of HIP with an "oldish" version of Kokkos.

Maybe you could try to bump your kokkos version to the latest in the 4 series (do not update to Kokkos 5, as this requires C++20)

From the idefix root directory:

cd src/kokkos
git pull
git checkout 4.7.01

[dutta-alankar]

I actually tried to switch to Kokkos 4.7.01 but the error still persists.
Should I try switching to an older ROCm?
Here's currently my loaded modules.
module purge && module load gcc/14 rocm/7.0 openmpi_gpu/5.0 fftw-mpi/3.3.10 hdf5-mpi/2.0.0

Error message with Kokkos 4.7.01

[ 43%] Building CXX object CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:288:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:31:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:923:19: error: static assertion failed due to requirement 'ViewMapping<Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::is_assignable': Incompatible View copy construction
  923 |     static_assert(Mapping::is_assignable,
      |                   ^~~~~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/output/scalarField.hpp:36:39: note: in instantiation of function template specialization 'Kokkos::View<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>::View<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   36 |       IdefixHostArray3D<real> arr3D = Kokkos::create_mirror(d3Darray);
      |                                       ^
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:288:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:202:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewMapping.hpp:2944:19: error: static assertion failed due to requirement 'is_assignable_space': View assignment must have compatible spaces
 2944 |     static_assert(is_assignable_space,
      |                   ^~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:925:14: note: in instantiation of member function 'Kokkos::Impl::ViewMapping<Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::assign' requested here
  925 |     Mapping::assign(m_map, rhs.m_map, rhs.m_track.m_tracker);
      |              ^
/ptmp/adutt/idefix-freya/src/output/scalarField.hpp:36:39: note: in instantiation of function template specialization 'Kokkos::View<double ***, Kokkos::LayoutRight, Kokkos::HostSpace>::View<double ***, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   36 |       IdefixHostArray3D<real> arr3D = Kokkos::create_mirror(d3Darray);
      |                                       ^
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:288:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:31:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:938:19: error: static assertion failed due to requirement 'ViewMapping<Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::is_assignable': Incompatible View copy assignment
  938 |     static_assert(Mapping::is_assignable, "Incompatible View copy assignment");
      |                   ^~~~~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/global.hpp:41:14: note: in instantiation of function template specialization 'Kokkos::View<int *, Kokkos::LayoutRight, Kokkos::HostSpace>::operator=<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   41 |   outArrHost = Kokkos::create_mirror_view(outArr);
      |              ^
/ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:633:25: note: in instantiation of function template specialization 'idfx::ConvertVectorToIdefixArray<int>' requested here
  633 |   this->mapVars = idfx::ConvertVectorToIdefixArray(mapVars);
      |                         ^
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:9:
In file included from /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.hpp:12:
In file included from /ptmp/adutt/idefix-freya/src/idefix.hpp:13:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Core.hpp:45:
In file included from /ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/decl/Kokkos_Declare_SERIAL.hpp:21:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial.hpp:288:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:20:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_Parallel.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/Kokkos_View.hpp:31:
In file included from /ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:202:
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewMapping.hpp:2944:19: error: static assertion failed due to requirement 'is_assignable_space': View assignment must have compatible spaces
 2944 |     static_assert(is_assignable_space,
      |                   ^~~~~~~~~~~~~~~~~~~
/ptmp/adutt/idefix-freya/src/kokkos/core/src/View/Kokkos_ViewLegacy.hpp:939:14: note: in instantiation of member function 'Kokkos::Impl::ViewMapping<Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::HostSpace>, Kokkos::ViewTraits<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>, void>::assign' requested here
  939 |     Mapping::assign(m_map, rhs.m_map, rhs.m_track.m_tracker);
      |              ^
/ptmp/adutt/idefix-freya/src/global.hpp:41:14: note: in instantiation of function template specialization 'Kokkos::View<int *, Kokkos::LayoutRight, Kokkos::HostSpace>::operator=<int *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HIPSpace>, Kokkos::Experimental::EmptyViewHooks>' requested here
   41 |   outArrHost = Kokkos::create_mirror_view(outArr);
      |              ^
/ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp:633:25: note: in instantiation of function template specialization 'idfx::ConvertVectorToIdefixArray<int>' requested here
  633 |   this->mapVars = idfx::ConvertVectorToIdefixArray(mapVars);
      |                         ^
4 errors generated when compiling for gfx942.
failed to execute:/mpcdf/soft/RHEL_9/packages/x86_64/rocm/7.0.2/lib/llvm/bin/clang++  --offload-arch=gfx942  -DKOKKOS_DEPENDENCE -DMHD=NO -DORDER=2 -DUSE_PROF_API=1 -DWITH_HDF5 -DWITH_MPI -D__HIP_PLATFORM_AMD__=1 -I/ptmp/adutt/idefix-freya/src/fluid/cooling -I/ptmp/adutt/idefix-freya/test/HD/trml -I/ptmp/adutt/idefix-freya/src/kokkos/core/src -I/ptmp/adutt/idefix-freya/src/dataBlock -I/ptmp/adutt/idefix-freya/src/dataBlock/planetarySystem -I/ptmp/adutt/idefix-freya/src/fluid -I/ptmp/adutt/idefix-freya/src/fluid/boundary -I/ptmp/adutt/idefix-freya/src/fluid/braginskii -I/ptmp/adutt/idefix-freya/src/fluid/constrainedTransport -I/ptmp/adutt/idefix-freya/src/fluid/eos -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver/HDsolvers -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver/MHDsolvers -I/ptmp/adutt/idefix-freya/src/fluid/RiemannSolver/Dustsolvers -I/ptmp/adutt/idefix-freya/src/fluid/tracer -I/ptmp/adutt/idefix-freya/src/output -I/ptmp/adutt/idefix-freya/src/rkl -I/ptmp/adutt/idefix-freya/src/gravity -I/ptmp/adutt/idefix-freya/src/utils -I/ptmp/adutt/idefix-freya/src/utils/iterativesolver -I/ptmp/adutt/idefix-freya/src -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/core/src -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/containers/src -I/ptmp/adutt/idefix-freya/src/kokkos/containers/src -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/algorithms/src -I/ptmp/adutt/idefix-freya/src/kokkos/algorithms/src -I/ptmp/adutt/idefix-freya/test/HD/trml/build/kokkos/simd/src -I/ptmp/adutt/idefix-freya/src/kokkos/simd/src -isystem /viper/u2/system/soft/RHEL_9/packages/znver4/openmpi_gpu/gcc_14-14.1.0-rocm_7.0-7.0.2/5.0.8/include -isystem /viper/u2/system/soft/RHEL_9/packages/znver4/openmpi_gpu/gcc_14-14.1.0-rocm_7.0-7.0.2/5.0.8/include/openmpi -isystem /mpcdf/soft/RHEL_9/packages/znver4/hdf5/gcc_14-14.1.0-openmpi_5.0-5.0.8/2.0.0/include -isystem /ptmp/adutt/idefix-freya/src/kokkos/tpls/desul/include -isystem /ptmp/adutt/idefix-freya/src/kokkos/tpls/mdspan/include -O3 -DNDEBUG -std=gnu++17 -fno-gpu-rdc -x hip -MD -MT CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o -MF CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o.d -o "CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o" -c /ptmp/adutt/idefix-freya/src/fluid/boundary/axis.cpp
make[2]: *** [CMakeFiles/idefix.dir/build.make:76: CMakeFiles/idefix.dir/src/fluid/boundary/axis.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:644: CMakeFiles/idefix.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

[glesur]

Can you try with Kokkos_ARCH_AMD_GFX942 in cmake?

[dutta-alankar]

@glesur Success this time following your suggestion and using Kokkos 4.7.01! The code compiles when I change Kokkos_ARCH_AMD_GFX942_APU to just Kokkos_ARCH_AMD_GFX942. Although the Kokkos manual tells not to do so (quite confusing).

On running the code, I get the following warning but the code runs nevertheless. Now, I don't have any reference to conclude if this can result in any penalty in performance.

Kokkos::HIP::initialize WARNING: running kernels for MI300X (discrete GPU) on a MI300A (APU).
Kokkos::HIP::initialize WARNING: running kernels for MI300X (discrete GPU) on a MI300A (APU).

[glesur]

fixed in #359