COMPASS

{{toc}}

h1. Install the platform without MAGMA

The COMPASS platform is distributed as a single bundle of CArMA and SuTrA libraries and YoGA and its AO extension for Yorick. 

h2. Hardware requirements

The system must contain at least an x86 CPU and a CUDA capable GPU. list of compatible GPUs can be found here http://www.nvidia.com/object/cuda_gpus.html. Specific requirements apply to clusters (to be updated).

h2. Environment requirements

The system must be running a 64 bit distribution of Linux or Mac OS with the latest NVIDIA drivers and "CUDA toolkit":https://developer.nvidia.com/cuda-downloads. The installation of the corresponding version of the "CULA tools":http://www.culatools.com/downloads/dense/ is also required. The following installation instructions are valid if the default installation paths have been selected for these components.

Additionally, to benefit from the user-oriented features of the platform, Yorick should be installed as well as the latest version of Python and the associated pygtk module. 

To install Yorick, download the latest version from the github repository:

<pre>

git clone https://github.com/dhmunro/yorick.git yorick.git

</pre>

then cd onto the created directory and install:

<pre>

./configure && make && make install

</pre>

once Yorick is locally installed, you will have to add this directory : yorick.git/relocate/bin to your PATH to have an easy access to the yorick executable. You may want to add support for command history by using rlwrap and alias the yorick executable as :

<pre>

alias yorick='rlwrap path_to_yorick_executable/yorick'

</pre>

h2. Installation process

First check out the latest version from the svn repository :

<pre>

svn co https://version-lesia.obspm.fr/repos/compass compass

</pre>

then go in the newly created directory and then trunk:

<pre>

cd compass/trunk

</pre>

once there, you need to modify system variables in the define_var.sh executable :

<pre>

emacs define_var.sh

</pre>

in this file define properly CUDA_ROOT, CULA_ROOT and YoGA path. Note that for the latter, as YoGA is distributed with SUTrA you should just point to the newly created trunk directory. On a Linux system you should normally have:

<pre>

export CUDA_ROOT=/usr/local/cuda

export CULA_ROOT=/usr/local/cula

export YOGA_DIR=/home/MyUserName/path2compass/trunk

</pre>

in this file, you also have to indicate the proper architecture of your GPU so as the compiler will generate the appropriate code. Modify the following line:

<pre>

export GENCODE="arch=compute_12,code=sm_12"

</pre>

and change both 12 to your architecture : for instance a Tesla Fermi will have 2.0 computing capabilities so change 12 to 20, a Kepler GPU will have 3.0 or 3.5 (K20) computing capabilities, change 12 to 30 (or 35).

Once this is done, you're ready to compile the whole library. First run define_var.sh to define the system variables that will be used during the compilation process:

<pre>

./define_var.sh

</pre>

then identify the absolute path to your Yorick executable using: 

<pre>

which yorick

</pre>

and run the compilation script:

<pre>

./reinstall absolute_path_to_yorick

</pre>

If you did not get any error, CArMA, SuTrA and YoGA are now installed on your machine. You can check that everything is working by launching a GUI to test a simulation:

<pre>

yorick -i yoga_ao/ywidgets/widget_ao.i

</pre>

h1. Install the platform with MAGMA

h2. Why MAGMA ?

The MAGMA project aims to develop a dense linear algebra library similar to LAPACK but for heterogeneous/hybrid architectures, starting with current "Multicore+GPU" systems.

Unlike CULA, MAGMA propose a dense linear algebra library handling double for free.

But MAGMA needs a LAPACK and a BLAS implementation. Actually, we try two options : ATLAS BLAS (free, easy to install) and MKL (free, need a registration but more powerful)

h2. Dependencies : gfortran

Use your package manager to install dependencies:

* on scientific linux : yum install gcc-gfortran libgfortran

* on debian : apt-get install gfortran gfortran-multilib

h2. Configure MAGMA with ATLAS

h3. Dependencies : blas, lapack, atlas

Use your package manager to install dependencies:

* on scientific linux : yum install blas-devel lapack-devel atlas-devel

* on debian : apt-get install libblas-dev liblapack-dev libatlas-base-dev libatlas-dev

The binary packages of ATLAS (and also OpenBLAS / GotoBLAS2) distributed by your Linux distribution (SL, Fedora, Debian,...) are generic packages, which are not optimized for a specific machine.

It is strongly advised to recompile ATLAS on your local machine to get best performances.

IMPORTANT NOTE: when building ATLAS, you must ensure that cpu throtling is disabled (if not timing measurement are erroneous, which may lead to an unoptimized build of ATLAS); see page

http://math-atlas.sourceforge.net/atlas_install/node5.html

h3. extraction

MAGMA is available here : http://icl.cs.utk.edu/magma/software/index.html

extract the tgz file and go into the new directory

> ~$ tar xf magma-1.4.1-beta.tar.gz

> ~$ cd magma-1.4.1

h3. configuration

You have to create your own make.inc :

* example on a scientific linux : *please verify GPU_TARGET, LAPACKDIR, ATLASDIR, CUDADIR*

<pre><code class="Makefile">

#//////////////////////////////////////////////////////////////////////////////

#   -- MAGMA (version 1.4.1) --

#      Univ. of Tennessee, Knoxville

#      Univ. of California, Berkeley

#      Univ. of Colorado, Denver

#      November 2013

#//////////////////////////////////////////////////////////////////////////////

# GPU_TARGET specifies for which GPU you want to compile MAGMA:

#     "Tesla"  (NVIDIA compute capability 1.x cards)

#     "Fermi"  (NVIDIA compute capability 2.x cards)

#     "Kepler" (NVIDIA compute capability 3.x cards)

# See http://developer.nvidia.com/cuda-gpus

GPU_TARGET ?= Fermi

CC        = gcc

NVCC      = nvcc

FORT      = gfortran

ARCH      = ar

ARCHFLAGS = cr

RANLIB    = ranlib

OPTS      = -fPIC -O3 -DADD_ -fopenmp -DMAGMA_SETAFFINITY

F77OPTS   = -fPIC -O3 -DADD_

FOPTS     = -fPIC -O3 -DADD_ -x f95-cpp-input

NVOPTS    =       -O3 -DADD_ -Xcompiler "-fno-strict-aliasing -fPIC"

LDOPTS    = -fPIC -fopenmp

# Depending on how ATLAS and LAPACK were compiled, you may need one or more of:

LIB       = -llapack -lf77blas -latlas -lcblas -lcublas -lcudart -lstdc++ -lm -lgfortran

# define library directories here or in your environment

LAPACKDIR = /usr/lib64

ATLASDIR  = /usr/lib64/atlas

CUDADIR   = /usr/local/cuda

LIBDIR    = -L$(LAPACKDIR) \

            -L$(ATLASDIR) \

            -L$(CUDADIR)/lib64

INC       = -I$(CUDADIR)/include

</code></pre>

* example on debian : *please verify GPU_TARGET, LAPACKDIR, ATLASDIR, CUDADIR*

<pre><code class="Makefile">

#//////////////////////////////////////////////////////////////////////////////

#   -- MAGMA (version 1.4.1) --

#      Univ. of Tennessee, Knoxville

#      Univ. of California, Berkeley

#      Univ. of Colorado, Denver

#      November 2013

#//////////////////////////////////////////////////////////////////////////////

# GPU_TARGET specifies for which GPU you want to compile MAGMA:

#     "Tesla"  (NVIDIA compute capability 1.x cards)

#     "Fermi"  (NVIDIA compute capability 2.x cards)

#     "Kepler" (NVIDIA compute capability 3.x cards)

# See http://developer.nvidia.com/cuda-gpus

GPU_TARGET ?= Fermi

CC        = gcc

NVCC      = nvcc

FORT      = gfortran

ARCH      = ar

ARCHFLAGS = cr

RANLIB    = ranlib

OPTS      = -fPIC -O3 -DADD_ -fopenmp -DMAGMA_SETAFFINITY

F77OPTS   = -fPIC -O3 -DADD_

FOPTS     = -fPIC -O3 -DADD_ -x f95-cpp-input

NVOPTS    =       -O3 -DADD_ -Xcompiler "-fno-strict-aliasing -fPIC" 

LDOPTS    = -fPIC -fopenmp

# Depending on how ATLAS and LAPACK were compiled, you may need one or more of:

LIB       = -llapack -lf77blas -latlas -lcblas -lcublas -lcudart -lstdc++ -lm -lgfortran

# define library directories here or in your environment

LAPACKDIR = /usr/lib

ATLASDIR  = /usr/lib

CUDADIR   = /usr/local/cuda

LIBDIR    = -L$(LAPACKDIR) \

            -L$(ATLASDIR) \

            -L$(CUDADIR)/lib64 \

            -L/usr/lib/x86_64-linux-gnu

INC       = -I$(CUDADIR)/include

</code></pre>

h2. Configure MAGMA with MKL

h3. extraction

To download MKL, you have to create a account here : https://registrationcenter.intel.com/RegCenter/NComForm.aspx?ProductID=1517

extract l_ccompxe_2013_sp1.1.106.tgz and go into l_ccompxe_2013_sp1.1.106

install it with ./install_GUI.sh and add IPP stuff to default choices

h3. configuration

* example on debian : *please verify GPU_TARGET, MKLROOT, CUDADIR*

<pre><code class="Makefile">

#//////////////////////////////////////////////////////////////////////////////

#   -- MAGMA (version 1.4.1-beta2) --

#      Univ. of Tennessee, Knoxville

#      Univ. of California, Berkeley

#      Univ. of Colorado, Denver

#      December 2013

#//////////////////////////////////////////////////////////////////////////////

# GPU_TARGET contains one or more of Tesla, Fermi, or Kepler,

# to specify for which GPUs you want to compile MAGMA:

#     Tesla  - NVIDIA compute capability 1.x cards

#     Fermi  - NVIDIA compute capability 2.x cards

#     Kepler - NVIDIA compute capability 3.x cards

# The default is all, "Tesla Fermi Kepler".

# See http://developer.nvidia.com/cuda-gpus

#

GPU_TARGET ?= Fermi

CC        = gcc

NVCC      = nvcc

FORT      = gfortran

ARCH      = ar

ARCHFLAGS = cr

RANLIB    = ranlib

OPTS      = -fPIC -O3 -DADD_ -Wall -fno-strict-aliasing -fopenmp -DMAGMA_WITH_MKL -DMAGMA_SETAFFINITY

F77OPTS   = -fPIC -O3 -DADD_ -Wall

FOPTS     = -fPIC -O3 -DADD_ -Wall -x f95-cpp-input

NVOPTS    =       -O3 -DADD_ -Xcompiler "-fno-strict-aliasing -fPIC"

LDOPTS    = -fPIC -fopenmp

# gcc with MKL 10.3, Intel threads

LIB       = -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread -lcublas -lcudart -lstdc++ -lm -liomp5 -lgfortran

# define library directories preferably in your environment, or here.

# for MKL run, e.g.: source /opt/intel/composerxe/mkl/bin/mklvars.sh intel64

MKLROOT ?= /opt/intel/composerxe/mkl

CUDADIR ?= /usr/local/cuda

-include make.check-mkl

-include make.check-cuda

LIBDIR    = -L$(MKLROOT)/lib/intel64 \

            -L$(CUDADIR)/lib64

INC       = -I$(CUDADIR)/include -I$(MKLROOT)/include

</code></pre>

In this example, I use gcc but with MKL, you can use icc instead of gcc. In this case, you have to compile yorick with icc. For this, you have to change the CC flag in Make.cfg  

h2. compilation and installation

h3. compilation

just compile the shared target (and test if you want)

> ~$ make -j 8 shared

h3. installation

To install libraries and include files in a given prefix, run:

> ~$ make install prefix=/usr/local/magma

The default prefix is /usr/local/magma. You can also set prefix in make.inc.

h3. tune (not tested)

For multi-GPU functions, set $MAGMA_NUM_GPUS to set the number of GPUs to use.

For multi-core BLAS libraries, set $OMP_NUM_THREADS or $MKL_NUM_THREADS or $VECLIB_MAXIMUM_THREADS to set the number of CPU threads, depending on your BLAS library.

h2. Platform installation

Just just define $MAGMA_PATH and use the standard procedure