How to convert vector to GPU compatible array?

[akashshinde1418]

Dear Team,

I have a problem that demands a vector to be passed inside the ParallelFor loop for GPU computations. Since I do not know the size of the vector at compile time, I cannot use Array1D/2D/3D or GpuArray. How can I solve this issue?

[atmyers]

Hi Akash,

You need to use amrex::DeviceVector, and capture the data pointer in your lambda function, as here.

[akashshinde1418]

Hi Andrew,
Thank you for the quick reply. I have checked your reference and see that the dataPtr() is used only for 1D vectors. Let us imagine a case where I'm dealing with a 3D vector. I have attached the code for the solution I could develop for this problem. I believe it is an inefficient way to tackle this problem. Can we have a solution where I don't have to explicitly copy the values from host to device rather just use a pointer to access the data from host?

` {
Vector< Vector< Vector< Real>>> Temp( 5, Vector< Vector< Real>>( 5, Vector< Real>( 5, 10.0)));

 for ( MFIter mfi( phi_old); mfi. isValid(); ++ mfi)
 {
 const Box& bx = mfi. validbox();

 amrex:: Gpu:: AsyncVector < Real> flag ( 125, 0.0);

 /*Operation to copy values from 3d Vector Temp to a 1d AsyncVector flag*/

 auto ptr = flag. dataPtr();

 amrex:: ParallelFor( bx, [=] AMREX_GPU_DEVICE( int i, int j, int k)
 {

      /*Do computations with ptr here*/

 });
 }

}`

[WeiqunZhang]

Something like this if you are Okay with Fortran array order (i.e., column major). If not, it's very easy to create your own version of amrex::Table3D.

Vector<Vector<Vector<Real>>> Temp(....);
Gpu::PinnedVector<Real> hv(5*5*5);
Gpu::DeviceVector<Real> dv(5*5*5);

// Copy data from Temp to hv on CPU

// Copy data to GPU
Gpu::copy(Gpu::hostToDevice, hv.begin(), hv.end(), dv.begin());

Table3D<Real> t3d(dv.data(), {0,0,0}, {5-1,5-1,5-1});

for (MFIter ...) {
    ParallelFor (...
    {
        Real value = t3d(2,3,4);  // i.e., Temp[4][3][2]
    });
}

If there is demand for row major ordering in amrex::Table3D, we can also add support in amrex.

[akashshinde1418]

Dear Weiqun Zhang Sir,
Thank you for the accurate response. For my problem, having a row-major ordering for Table3D will be highly beneficial. Are there any immediate plans to add the support?

[WeiqunZhang]

We can add that in the future. For now, you can just do like what's in the example above. That is t3d(r,s,t) is equivalent to Temp[t][s][r].

[WeiqunZhang]

Row-major order is added in #3973

[akashshinde1418]

Thank you for adding the support!