Install the platform » Historique » Révision 28
Révision 27 (Arnaud Sevin, 23/09/2014 14:30) → Révision 28/62 (Arnaud Sevin, 10/11/2015 14:08)
{{toc}}
h1. Install the platform without MAGMA
The COMPASS platform is distributed as a single bundle of CArMA and SuTrA libraries and YoGA and its AO extension for Yorick.
h2. Hardware requirements
The system must contain at least an x86 CPU and a CUDA capable GPU. list of compatible GPUs can be found here http://www.nvidia.com/object/cuda_gpus.html. Specific requirements apply to clusters (to be updated).
h2. Environment requirements
The system must be running a 64 bit distribution of Linux or Mac OS with the latest NVIDIA drivers and "CUDA toolkit":https://developer.nvidia.com/cuda-downloads. The installation of the corresponding version of the "CULA tools":http://www.culatools.com/downloads/dense/ is also required. The following installation instructions are valid if the default installation paths have been selected for these components.
Additionally, to benefit from the user-oriented features of the platform, Anaconda Yorick should be installed. installed as well as the latest version of Python and the associated pygtk module.
To install Yorick, download the latest version from the github repository:
In <pre>
git clone https://github.com/dhmunro/yorick.git yorick.git
</pre>
then cd onto the last versions of compass (r608+), created directory and install:
<pre>
./configure && make && make install
</pre>
once Yorick is no more supported.
For the widget, locally installed, you also need pyQTgraph. You can install it like will have to add this directory :
yorick.git/relocate/bin to your PATH to have an easy access to the yorick executable. You may want to add support for command history by using rlwrap and alias the yorick executable as :
<pre>
pip install pyqtgraph
alias yorick='rlwrap path_to_yorick_executable/yorick'
</pre>
h2. Installation process
First check out the latest version from the svn repository :
<pre>
svn co https://version-lesia.obspm.fr/repos/compass compass
</pre>
then go in the newly created directory and then trunk:
<pre>
cd compass/trunk
</pre>
once there, you need to modify system variables in our .bashrc the define_var.sh executable :
<pre>
# CUDA default definitions emacs define_var.sh
export CUDA_ROOT=$CUDA_ROOT #/usr/local/cuda </pre>
export CUDA_INC_PATH=$CUDA_ROOT/include in this file define properly CUDA_ROOT, CULA_ROOT and YoGA path. Note that for the latter, as YoGA is distributed with SUTrA you should just point to the newly created trunk directory. On a Linux system you should normally have:
export CUDA_LIB_PATH=$CUDA_ROOT/lib <pre>
export CUDA_LIB_PATH_64=$CUDA_ROOT/lib64 CUDA_ROOT=/usr/local/cuda
export CPLUS_INCLUDE_PATH=$CUDA_INC_PATH CULA_ROOT=/usr/local/cula
export PATH=$CUDA_ROOT/bin:$PATH YOGA_DIR=/home/MyUserName/path2compass/trunk
</pre>
in this file, you also have to indicate the proper architecture of your GPU so as the compiler will generate the appropriate code. Modify the following line:
<pre>
export GENCODE="arch=compute_12,code=sm_12"
</pre>
and change both 12 to your architecture : for instance a Tesla Fermi will have 2.0 computing capabilities so change 12 to 20, a Kepler GPU will have 3.0 or 3.5 (K20) computing capabilities, change 12 to 30 (or 35).
If you are using CULA, you have Once this is done, you're ready to specify it: compile the whole library. First run define_var.sh to define the system variables that will be used during the compilation process:
<pre>
# CULA default definitions ./define_var.sh
export CULA_ROOT= /usr/local/cula
export CULA_INC_PATH= $CULA_ROOT/include
export CULA_LIB_PATH= $CULA_ROOT/lib
export CULA_LIB_PATH_64= $CULA_ROOT/lib64
</pre>
If you are using MAGMA, you have then identify the absolute path to specify it:
your Yorick executable using:
<pre>
# MAGMA definitions (uncomment this line if MAGMA is installed) which yorick
export MAGMA_ROOT=/usr/local/magma
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MAGMA_ROOT/lib
export PKG_CONFIG_PATH=$MAGMA_ROOT/lib/pkgconfig
</pre>
Last variables to define:
<pre>
export COMPASS_ROOT=/path/to/compass/trunk
export NAGA_ROOT=$COMPASS_ROOT/naga
export SHESHA_ROOT=$COMPASS_ROOT/shesha
export LD_LIBRARY_PATH=$COMPASS_ROOT/libcarma:$COMPASS_ROOT/libsutra:$CUDA_LIB_PATH_64:$CUDA_LIB_PATH:$CULA_LIB_PATH_64:$CULA_LIB_PATH:$LD_LIBRARY_PATH
</pre>
Once this is done, you're ready to compile and run the whole library: compilation script:
<pre>
make clean install ./reinstall absolute_path_to_yorick
</pre>
If you did not get any error, CArMA, SuTrA, NAGA SuTrA and SHESHA YoGA are now installed on your machine. You can check that everything is working by launching a GUI to test a simulation:
<pre>
cd $SHESHA_ROOT/widgets && ipython yorick -i widget_ao.py yoga_ao/ywidgets/widget_ao.i
</pre>
h1. Install the platform with MAGMA
h2. Why MAGMA ?
The MAGMA project aims to develop a dense linear algebra library similar to LAPACK but for heterogeneous/hybrid architectures, starting with current "Multicore+GPU" systems.
Unlike CULA, MAGMA propose a dense linear algebra library handling double for free.
But MAGMA needs a LAPACK and a BLAS implementation. Actually, we try two options : ATLAS BLAS (free, easy to install) and MKL (free, need a registration but more powerful)
h2. Dependencies : gfortran
Use your package manager to install dependencies:
* on scientific linux : yum install gcc-gfortran libgfortran
* on debian : apt-get install gfortran gfortran-multilib
h2. Configure MAGMA with openBLAS ATLAS
h3. Dependencies : openblas (http://www.openblas.net) blas, lapack, atlas
First, clone the GIT repository: Use your package manager to install dependencies:
<pre> * on scientific linux : yum install blas-devel lapack-devel atlas-devel
git clone https://github.com/xianyi/OpenBLAS.git * on debian : apt-get install libblas-dev liblapack-dev libatlas-base-dev libatlas-dev
The binary packages of ATLAS (and also OpenBLAS / GotoBLAS2) distributed by your Linux distribution (SL, Fedora, Debian,...) are generic packages, which are not optimized for a specific machine.
</pre> It is strongly advised to recompile ATLAS on your local machine to get best performances.
compile it: debian easy method :
<pre> sudo apt-get build-dep atlas
sudo apt-get install build-essential devscripts lintian diffutils patch patchutils
apt-get source atlas
cd OpenBLAS/ atlas-*
make fakeroot debian/rules custom
</pre> cd ..
ls libatlas*.deb
install it: Then, for each of the entries listed by the ls command (there may be a quicker way to do it, using "*"), type:
<pre>
sudo make dpkg -i <filename here>.deb
This should install PREFIX=/usr/local/openblas-haswellp-r0.2.14.a
</pre> an optimised build of the version of ATLAS that Ubuntu provides. There's a more recent version, but it doesn't come with the easy "debian/rules" stuff, so this method won't work.
add IMPORTANT NOTE: when building ATLAS, you must ensure that cpu throtling is disabled (if not timing measurement are erroneous, which may lead to you .bashrc: an unoptimized build of ATLAS); see page
<pre>
export OPENBLAS_ROOT=/usr/local/openblas-haswellp-r0.2.14.a
</pre>
http://math-atlas.sourceforge.net/atlas_install/node5.html
h3. extraction
MAGMA is available here : http://icl.cs.utk.edu/magma/software/index.html
extract the tgz file and go into the new directory
> ~$ tar xf magma-1.7.0-b.tar.gz magma-1.4.1-beta.tar.gz
> ~$ cd magma-1.7.0 magma-1.4.1
h3. configuration
You have to create your own make.inc based on make.inc.openblas: :
* example on a scientific linux : *please verify GPU_TARGET, LAPACKDIR, ATLASDIR, CUDADIR*
<pre><code class="Makefile">
#//////////////////////////////////////////////////////////////////////////////
# -- MAGMA (version 1.7.0) 1.4.1) --
# Univ. of Tennessee, Knoxville
# Univ. of California, Berkeley
# Univ. of Colorado, Denver
# @date September 2015 November 2013
#//////////////////////////////////////////////////////////////////////////////
# GPU_TARGET contains one or more of Tesla, Fermi, or Kepler,
# to specify specifies for which GPUs GPU you want to compile MAGMA:
# Tesla "Tesla" - NVIDIA (NVIDIA compute capability 1.x cards (no longer supported in CUDA 6.5) cards)
# Fermi "Fermi" - NVIDIA (NVIDIA compute capability 2.x cards cards)
# Kepler - NVIDIA "Kepler" (NVIDIA compute capability 3.x cards cards)
# The default is "Fermi Kepler".
# See http://developer.nvidia.com/cuda-gpus
#
GPU_TARGET ?= Kepler Fermi
# --------------------
# programs
CC = gcc
CXX = g++
NVCC = nvcc
FORT = gfortran
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
OPTS = -fPIC -O3 -DADD_ -fopenmp -DMAGMA_SETAFFINITY
F77OPTS = -fPIC -O3 -DADD_
FOPTS = -fPIC -O3 -DADD_ -x f95-cpp-input
NVOPTS = -O3 -DADD_ -Xcompiler "-fno-strict-aliasing -fPIC"
LDOPTS = -fPIC -fopenmp
# -------------------- Depending on how ATLAS and LAPACK were compiled, you may need one or more of:
LIB = -llapack -lf77blas -latlas -lcblas -lcublas -lcudart -lstdc++ -lm -lgfortran
# flags define library directories here or in your environment
LAPACKDIR = /usr/lib64
ATLASDIR = /usr/lib64/atlas
CUDADIR = /usr/local/cuda
LIBDIR = -L$(LAPACKDIR) \
-L$(ATLASDIR) \
-L$(CUDADIR)/lib64
INC = -I$(CUDADIR)/include
</code></pre>
* example on debian : *please verify GPU_TARGET, LAPACKDIR, ATLASDIR, CUDADIR*
<pre><code class="Makefile">
#//////////////////////////////////////////////////////////////////////////////
# Use -fPIC -- MAGMA (version 1.4.1) --
# Univ. of Tennessee, Knoxville
# Univ. of California, Berkeley
# Univ. of Colorado, Denver
# November 2013
#//////////////////////////////////////////////////////////////////////////////
# GPU_TARGET specifies for which GPU you want to make shared (.so) and static (.a) library; compile MAGMA:
# can be commented out if making only static library. "Tesla" (NVIDIA compute capability 1.x cards)
FPIC # "Fermi" (NVIDIA compute capability 2.x cards)
# "Kepler" (NVIDIA compute capability 3.x cards)
# See http://developer.nvidia.com/cuda-gpus
GPU_TARGET ?= Fermi
CC = gcc
NVCC = -fPIC nvcc
FORT = gfortran
CFLAGS ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
OPTS = -fPIC -O3 $(FPIC) -DADD_ -Wall -fopenmp
FFLAGS -DMAGMA_SETAFFINITY
F77OPTS = -fPIC -O3 $(FPIC) -DADD_ -Wall -Wno-unused-dummy-argument
F90FLAGS FOPTS = -fPIC -O3 $(FPIC) -DADD_ -Wall -Wno-unused-dummy-argument -x f95-cpp-input
NVCCFLAGS NVOPTS = -O3 -DADD_ -Xcompiler "$(FPIC)"
LDFLAGS "-fno-strict-aliasing -fPIC"
LDOPTS = $(FPIC) -fPIC -fopenmp
# --------------------
# libraries
# gcc with OpenBLAS (includes LAPACK) Depending on how ATLAS and LAPACK were compiled, you may need one or more of:
LIB = -lopenblas
LIB += -llapack -lf77blas -latlas -lcblas -lcublas -lcudart
# --------------------
# directories -lstdc++ -lm -lgfortran
# define library directories preferably here or in your environment, or here. environment
OPENBLASDIR LAPACKDIR = /usr/local/openblas-haswellp-r0.2.14.a /usr/lib
ATLASDIR = /usr/lib
CUDADIR = /usr/local/cuda
-include make.check-openblas
-include make.check-cuda
LIBDIR = -L$(LAPACKDIR) \
-L$(ATLASDIR) \
-L$(CUDADIR)/lib64 \
-L$(OPENBLASDIR)/lib -L/usr/lib/x86_64-linux-gnu
INC = -I$(CUDADIR)/include
</code></pre>
h2. Configure MAGMA with MKL
h3. extraction
To download MKL, you have to create a account here : https://registrationcenter.intel.com/RegCenter/NComForm.aspx?ProductID=1517
extract l_ccompxe_2013_sp1.1.106.tgz and go into l_ccompxe_2013_sp1.1.106
install it with ./install_GUI.sh and add IPP stuff to default choices
h3. configuration
You have to create your own make.inc based * example on make.inc.mkl-gcc-ilp64:
example: debian : *please verify GPU_TARGET, MKLROOT, CUDADIR*
<pre><code class="Makefile">
#//////////////////////////////////////////////////////////////////////////////
# -- MAGMA (version 1.7.0) 1.4.1-beta2) --
# Univ. of Tennessee, Knoxville
# Univ. of California, Berkeley
# Univ. of Colorado, Denver
# @date September 2015 December 2013
#//////////////////////////////////////////////////////////////////////////////
# GPU_TARGET contains one or more of Tesla, Fermi, or Kepler,
# to specify for which GPUs you want to compile MAGMA:
# Tesla - NVIDIA compute capability 1.x cards (no longer supported in CUDA 6.5)
# Fermi - NVIDIA compute capability 2.x cards
# Kepler - NVIDIA compute capability 3.x cards
# The default is "Fermi all, "Tesla Fermi Kepler".
# See http://developer.nvidia.com/cuda-gpus
#
#GPU_TARGET GPU_TARGET ?= Fermi Kepler
# --------------------
# programs
CC = gcc
CXX = g++
NVCC = nvcc
FORT = gfortran
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
# --------------------
# flags
# Use -fPIC to make shared (.so) and static (.a) library;
# can be commented out if making only static library.
FPIC OPTS = -fPIC
CFLAGS = -O3 $(FPIC) -DADD_ -Wall -Wshadow -fno-strict-aliasing -fopenmp -DMAGMA_WITH_MKL -DMAGMA_SETAFFINITY
FFLAGS F77OPTS = -fPIC -O3 $(FPIC) -DADD_ -Wall -Wno-unused-dummy-argument
F90FLAGS FOPTS = -fPIC -O3 $(FPIC) -DADD_ -Wall -Wno-unused-dummy-argument -x f95-cpp-input
NVCCFLAGS NVOPTS = -O3 -DADD_ -Xcompiler "$(FPIC) -Wall -Wno-unused-function" "-fno-strict-aliasing -fPIC"
LDFLAGS LDOPTS = $(FPIC) -fPIC -fopenmp
# Defining MAGMA_ILP64 or MKL_ILP64 changes magma_int_t to int64_t in include/magma_types.h
CFLAGS += -DMKL_ILP64
FFLAGS += -fdefault-integer-8
F90FLAGS += -fdefault-integer-8
NVCCFLAGS += -DMKL_ILP64
# Options to do extra checks for non-standard things like variable length arrays;
# it is safe to disable all these
CFLAGS += -pedantic -Wno-long-long
#CFLAGS += -Werror # uncomment to ensure all warnings are dealt with
CXXFLAGS := $(CFLAGS) -std=c++98
CFLAGS += -std=c99
# --------------------
# libraries
# IMPORTANT: this link line is for 64-bit int !!!!
# For regular 64-bit builds using 64-bit pointers and 32-bit int,
# use the lp64 library, not the ilp64 library. See make.inc.mkl-gcc or make.inc.mkl-icc.
# see MKL Link Advisor at http://software.intel.com/sites/products/mkl/
# gcc with MKL 10.3, Intel threads, 64-bit int threads
# note -DMAGMA_ILP64 or -DMKL_ILP64, and -fdefault-integer-8 in FFLAGS above
LIB = -lmkl_intel_ilp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread -lcublas -lcudart -lstdc++ -lm -liomp5 -lgfortran
LIB += -lcublas -lcudart
# --------------------
# directories
# define library directories preferably in your environment, or here.
# for MKL run, e.g.: source /opt/intel/composerxe/mkl/bin/mklvars.sh intel64
#MKLROOT MKLROOT ?= /opt/intel/composerxe/mkl
#CUDADIR CUDADIR ?= /usr/local/cuda
-include make.check-mkl
-include make.check-cuda
LIBDIR = -L$(CUDADIR)/lib64 -L$(MKLROOT)/lib/intel64 \
-L$(MKLROOT)/lib/intel64 -L$(CUDADIR)/lib64
INC = -I$(CUDADIR)/include \
-I$(MKLROOT)/include
</code></pre>
In this example, I use gcc but with MKL, you can use icc instead of gcc. In this case, you have to compile yorick with icc. For this, you have to change the CC flag in Make.cfg
h2. compilation and installation
h3. compilation
just compile the shared target (and test if you want)
> ~$ make -j 8 shared sparse
h3. installation
To install libraries and include files in a given prefix, run:
> ~$ make install prefix=/usr/local/magma
The default prefix is /usr/local/magma. You can also set prefix in make.inc.
h3. tune (not tested)
For multi-GPU functions, set $MAGMA_NUM_GPUS to set the number of GPUs to use.
For multi-core BLAS libraries, set $OMP_NUM_THREADS or $MKL_NUM_THREADS or $VECLIB_MAXIMUM_THREADS to set the number of CPU threads, depending on your BLAS library.
h2. Platform installation
Just just define $MAGMA_PATH and use the standard procedure