Skip to content

Matrix multiplication on GPUs for matrices stored on a CPU. Similar to cublasXt, but ported to both NVIDIA and AMD GPUs.

License

Notifications You must be signed in to change notification settings

eth-cscs/Tiled-MM

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Table of Contents

Overview

Tiled-MM is a very fast and easy-to-use library for multiplying matrices on GPU. As opposed to NVIDIA's cublas, this library takes pointer from the host side (CPU), splits the matrices into tiles, pipelines them efficiently to the GPU and copies the result back to the CPU. It can serve as almost a drop-in replacement for cublasXt, and is ported to both NVIDIA and AMD gpus.

It offers more features than the standard cublas API. For example, the user can specify the number of gpu streams to be used, as well as the tile size for each dimension separately, which is not possible with the standard cublas API.

Tiled-MM is used in production as a backend of the COSMA algorithm and is thus well-tested.

Performance

The benchmarks were performed on a single node of Piz Daint Supercomputer (Cray XC50), equipped with a P100 NVIDIA GPU. We compared the performance of our library Tiled-MM with the vanilla version of cublasXt and also with the manually tuned version of cublasXt, where we manually set the tile size to 4000 and enabled the pinned memory mode. Tiled-MM was substantially faster than the vanilla version of cublasXt, and achieved similar performance as the manually tuned version of cublasXt, as can be seen from the results below.

In the benchmark, we used double precision, square matrices given in column-major ordering, and alpha = beta = 1.0.

Features:

  • The user can specify the tile size of each dimension separately.
  • The user can specify the number of streams to be used.
  • The user can reuse the same context (and thus the same device memory) for many multiplications which can lead to significant performance improvements.
  • Fully templatized, supporting arbitrary data types.
  • Ported to both NVIDIA and AMD GPUs.

Building and Installing

Assuming that you want to use the gcc 8 compiler, you can build the project as follows:

# clone the repo
git clone https://github.com/eth-cscs/Tiled-MM
cd Tiled-MM
mkdir build
cd build

# build
CC=gcc-8 CXX=g++-8 cmake -DTILEDMM_GPU_BACKEND=CUDA ..

# compile
make -j 4

When building the examples cxxopts is required. It is available in most package manager, apt-get install libcxxopts-dev (ubuntu) or brew install cxxopts (macos).

The option -DTILEDMM_GPU_BACKEND can have the following values:

  • CUDA: for NVIDIA GPUs
  • ROCM: for AMD GPUs

Minimal Working Example

Using the library is very simple, just include #include <tiled_mm.hpp> and use it as follows:

// A dimensions: m x k
auto a_host = gpu::malloc_pinned<double>(m * k, 1);
// B dimensions: k x n
auto b_host = gpu::malloc_pinned<double>(k * n, 1);
// C dimensions: m x n
auto c_host = gpu::malloc_pinned<double>(m * n, 0);

double alpha = 1.0;
double beta = 0.0;

// preallocates device buffers and other CUDA stuff
// the context does not have to be created explicitly
// so the user can omit this part
auto ctx = gpu::make_context();

// compute c = alpha * a * b + beta * c
// There is also a version without ctx, in case the user
// does not want to create the context explicitly
gpu::gemm(*ctx,
          trans_a, trans_b,
          m, n, k,
          alpha,
          a_host, ld_a,
          b_host, ld_b,
          beta,
          c_host, ld_c);

// optionally, we can set the following two boolean flags
bool pin_buffers = false; // since a_host, b_host and c_host are already pinned, gpu::dgemm should not pin them
bool copy_c_back = true;  // if we want to copy the result back to the host or leave it on the gpu
gpu::gemm(*ctx,
          trans_a, trans_b,
          m, n, k,
          alpha,
          a_host, ld_a,
          b_host, ld_b,
          beta,
          c_host, ld_c,
          pin_buffers, copy_c_back);

// if copy_c_back == false, the result is stored on the device with the following pointer:
double* c_device = ctx->get_full_device_buffer_c().data()

When creating the context, the user can specify tile dimensions and the number of streams to be used as:

int tile_size_m = 5000;
int tile_size_n = 5000;
int tile_size_k = 5000;
int n_streams = 2;

auto ctx = gpu::make_context(n_streams, tile_size_m, tile_size_n, tile_size_k);

Running the Benchmarks

For detailed benchmarking, there is a miniapp that takes the host pointers for A, B and C and computes C = beta * C + alpha * A * B outputing the time-to-solution, as well as the throughput.

The miniapp consists of the executable ./build/examples/multiply which can be run with the following command line (assuming we are in the root folder of the project):

./build/examples/multiply -m 10000 -n 10000 -k 10000 -r 1

The overview of all supported options is given below:

Option Flags POSSIBLE VALUES DESCRIPTION
m (--m_dim) positive integer Number of rows of C
n (--n_dim) positive integer Number of columns of C
k (--k_dim) positive integer size of the shared dimension between matrices A and B
--tile_m positive integer tile size for dimension m
--tile_n positive integer tile size for dimension n
--tile_k positive integer tile size for dimension k
--ld_a positive integer leading dimension of matrix A
--ld_b positive integer leading dimension of matrix B
--ld_c positive integer leading dimension of matrix C
-t (--transpose) a string XY, where X, Y can be one of {N, T, C} transpose flags for matrices A and B
--alpha real value (double) the alpha in C = beta * C + alpha * A * B
--beta real value (double) the beta in C = beta * C + alpha * A * B

For example, running with the following flags:

./build/examples/multiply -m 1000 -n 1000 -k 1000 --transpose=TN -r 1

should produce the following output:

==================================================
                Benchmarking Tiled-MM
==================================================
         MATRIX SIZES
=============================
 A = (1000, 1000)
 B = (1000, 1000)
 C = (1000, 1000)
=============================
         LEADING DIMS
=============================
 LD_A = 1000
 LD_B = 1000
 LD_C = 1000
=============================
      SCALING CONSTANTS
=============================
 alpha = 1
 beta  = 1
=============================
      TRANSPOSE FLAGS
=============================
 trans_a = T
 trans_b = N
=============================
         TILE SIZES
=============================
 tile_m = 5000
 tile_n = 5000
 tile_k = 5000
=============================
      ADDITIONAL OPTIONS
=============================
 num. of gpu streams = 2
 num. of repetitions = 1
=============================

==================================================
         Results of benchmarking Tiled-MM
==================================================
 1) The version with copying C to back to host:
    -> Avg Time [ms] = 11
    -> Throughput [Gflops] = 181.818
==================================================
 2) The version without copying C to back to host:
    -> Avg Time [ms] = 10
    -> Throughput [Gflops] = 200
==================================================

Testing

For testing purposes, there is a testing miniapp that generates random matrices A, B and C, computes C = beta * C + alpha * A * B with Tiled-MM as well as with blas and outputs whether the results are correct.

The miniapp consists of the executable ./build/tests/test-multiply supports the same parameters as the benchmarking miniapp (see above). It can be run e.g. with the following command line (assuming we are in the root folder of the project):

./build/tests/test-multiply -m 1000 -n 1000 -k 1000 --transpose=TN

which should produce the following output:

==================================================
                Benchmarking Tiled-MM
==================================================
         MATRIX SIZES
=============================
 A = (1000, 1000)
 B = (1000, 1000)
 C = (1000, 1000)
=============================
         LEADING DIMS
=============================
 LD_A = 1000
 LD_B = 1000
 LD_C = 1000
=============================
      SCALING CONSTANTS
=============================
 alpha = 1
 beta  = 1
=============================
      TRANSPOSE FLAGS
=============================
 trans_a = T
 trans_b = N
=============================
         TILE SIZES
=============================
 tile_m = 5000
 tile_n = 5000
 tile_k = 5000
=============================
      ADDITIONAL OPTIONS
=============================
 num. of gpu streams = 2
 num. of repetitions = 1
=============================
Time [ms] with copying C back: 11
Time [ms] without copying C back: 10
The result is CORRECT

Running make test will few default tests.

Author

Marko Kabic (marko.kabic@inf.ethz.ch)