Scope

Scope may be downloaded from https://github.com/c3sr/scope/releases

master

A benchmark framework developed by the IBM-ILLINOIS Center for Cognitive Computing Systems Research (C3SR) in collaboration with the IMPACT group at the University of Illinois.

Primary maintainers:

• Carl Pearson (web) (email)
• Abdul Dakkak (email)
• Cheng Li (web) (email)

Project Advisors:

• Prof. Wen-mei Hwu (UofI)
• Dr. Jinjun Xiong (IBM Research)

Various benchmark suites using Scope are under development:

• Comm|Scope - CUDA/NUMA data transfer performance (Carl Pearson, UIUC)
• NCCL|Scope - GPU collective communication performance (Sarah Hashash, UIUC)
• Histo|Scope - CUDA histogram techniques (Carl Pearson, UIUC)
• DDL|Scope - IBM Distributed Deep Learning Library benchmarks (Vandana Kulkarni, UIUC)
• TCU|Scope - CUDA/TCU performance primitives (Abdul Dakkak, UIUC)
• FrameworkLayer|Scope - Evaluation of neural network layers across frameworks (Cheng Li and Abdul Dakkak, UIUC)
• CUDNN|Scope - Evaluation of neural network layers using CuDNN(Cheng Li and Abdul Dakkak, UIUC)
• Misc|Scope - experimental or miscellaneous benchmarks

Quick Start

• Install CMake 3.12+
• clone, checkout the lastest release, update submodules to match, and build

  git clone https://github.com/c3sr/scope.git --recursive
  cd scope
  git checkout v1.3.2                 # or the latest, `git tag --list`
  git submodule update                # match benchmark versions
  mkdir build && cd build
  cmake .. -DENABLE_COMM=ON           # or other scopes
  make -j`nproc`
  ./scope --benchmark_list_tests=true # list all scopesCopy to clipboardErrorCopied

Install CMake >= 3.12

User install of CMake 3.12 (preferred)

If your system has CMake < 3.12, we suggest installing CMake 3.12+ in the user's $HOME directory.

On x86-64, the following will download CMake 3.12.0 and install it in $HOME/software/cmake-3.12.0.

cd /tmp
wget https://cmake.org/files/v3.12/cmake-3.12.0-Linux-x86_64.sh
mkdir -p $HOME/software/cmake-3.12.0
sudo sh cmake-3.12.0-Linux-x86_64.sh --prefix=$HOME/software/cmake-3.12.0 --exclude-subdirCopy to clipboardErrorCopied

You will then need to add $HOME/software/cmake-3.12.0/bin</code> to your path.<br>For many linux users, you add this to your <code>$HOME/.bashrc:

export PATH="$PATH:$HOME/software/cmake-3.12.0/bin"`Copy to clipboardErrorCopied

On ppc64le, you will need to download the CMake source from the CMake website and build it.

System install of CMake 3.12

If you don't already know how to do this before reading, this is probably not the right option for you.
First, uninstall any existing system install of CMake.
Then, follow the User install instructions above, but choose a system prefix for the installation.

Compile

To compile the project run the following commands (making sure nvcc is in your $PATH, which is typically at /usr/local/cuda/bin/nvcc)

git clone https://github.com/c3sr/scope.git --recursive
cd scope
mkdir -p build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
makeCopy to clipboardErrorCopied

The build system uses Hunter to download all dependencies.
If you have trouble downloading dependencies, check to make sure Hunter/CMake can use SSL.
Or you can forego Hunter entirely and provide your own dependencies.

You will need to enable the particular scopes that provide the benchmarks you want to run

if you get errors about nvcc not supporting your gcc compiler, then you may want to use

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_HOST_COMPILER=`which gcc-6` .. Copy to clipboardErrorCopied

You can optionally choose your own CUDA archs that you would like to be compiled:

cmake -DNVCC_ARCH_FLAGS="2.0 2.1 2.0 2.1 3.0 3.2 3.5 3.7 5.0 5.2 5.3" ..Copy to clipboardErrorCopied

The accepted syntax is the same as the CUDA_SELECT_NVCC_ARCH_FLAGS syntax in the FindCUDA module.

You can disable or enable individual scopes

cmake -DENABLE_MISC=0 ...Copy to clipboardErrorCopied

The submodules should automatically be checked out.
If not, try checking them out yourself:

git submodule update --init --recursiveCopy to clipboardErrorCopied

or to update modules to the proper verions

git submodule update --recursive --remoteCopy to clipboardErrorCopied

Available Benchmarks

The available benchmarks and descriptions are listed here. You can list all the benchmarks with

./scope --benchmark_list_tests=trueCopy to clipboardErrorCopied

you can filter the benchmarks that are run with a regular expression passed to --benchmark_filter.

./scope --benchmark_filter=[regex]Copy to clipboardErrorCopied

for example

./scope --benchmark_filter=SGEMMCopy to clipboardErrorCopied

futher controls over the benchmarks are explained in the --help option

Run all the benchmarks

This is not generally recommended, as it will take quite some time.

./scopeCopy to clipboardErrorCopied

The above will output to stdout something like

------------------------------------------------------------------------------
Benchmark                       Time           CPU Iterations UserCounters...
------------------------------------------------------------------------------
SGEMM/1000/1/1/-1/1             5 us          5 us     126475 K=1 M=1000 N=1 alpha=-1 beta=1
SGEMM/128/169/1728/1/0        539 us        534 us       1314 K=1.728k M=128 N=169 alpha=1 beta=0
SGEMM/128/729/1200/1/0       1042 us       1035 us        689 K=1.2k M=128 N=729 alpha=1 beta=0
SGEMM/192/169/1728/1/0        729 us        724 us        869 K=1.728k M=192 N=169 alpha=1 beta=0
SGEMM/256/169/1/1/1             9 us          9 us      75928 K=1 M=256 N=169 alpha=1 beta=1
SGEMM/256/729/1/1/1            35 us         35 us      20285 K=1 M=256 N=729 alpha=1 beta=1
SGEMM/384/169/1/1/1            18 us         18 us      45886 K=1 M=384 N=169 alpha=1 beta=1
SGEMM/384/169/2304/1/0       2475 us       2412 us        327 K=2.304k M=384 N=169 alpha=1 beta=0
SGEMM/50/1000/1/1/1            10 us         10 us      73312 K=1 M=50 N=1000 alpha=1 beta=1
SGEMM/50/1000/4096/1/0       6364 us       5803 us        100 K=4.096k M=50 N=1000 alpha=1 beta=0
SGEMM/50/4096/1/1/1            46 us         45 us      13491 K=1 M=50 N=4.096k alpha=1 beta=1
SGEMM/50/4096/4096/1/0      29223 us      26913 us         20 K=4.096k M=50 N=4.096k alpha=1 beta=0
SGEMM/50/4096/9216/1/0      55410 us      55181 us         10 K=9.216k M=50 N=4.096k alpha=1 beta=0
SGEMM/96/3025/1/1/1            55 us         51 us      14408 K=1 M=96 N=3.025k alpha=1 beta=1
SGEMM/96/3025/363/1/0        1313 us       1295 us        570 K=363 M=96 N=3.025k alpha=1 beta=0Copy to clipboardErrorCopied

Output as JSON using

./scope --benchmark_out_format=json --benchmark_out=test.jsonCopy to clipboardErrorCopied

or preferably

./scope --benchmark_out_format=json --benchmark_out=`hostname`.jsonCopy to clipboardErrorCopied

Repeat benchmark runs with

./scope --benchmark_repetitions=5Copy to clipboardErrorCopied

Plot Benchmark JSON files

Try the ScopePlot python package.

pip install scope_plotCopy to clipboardErrorCopied

On Minsky With PowerAI

cd build && rm -fr * && OpenBLAS=/opt/DL/openblas cmake -DCMAKE_BUILD_TYPE=Release .. -DOpenBLAS=/opt/DL/openblasCopy to clipboardErrorCopied

Disable CPU frequency scaling

If you see this error:

***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.Copy to clipboardErrorCopied

you might want to disable the CPU frequency scaling while running the benchmark.
On ubuntu, install

apt install linux-tools-$(uname -r)Copy to clipboardErrorCopied

then

sudo cpupower frequency-set --governor performance
./scope
sudo cpupower frequency-set --governor powersaveCopy to clipboardErrorCopied

Run with Docker

Install nvidia-docker, then, list the available benchmarks.

nvidia-docker run  --rm raiproject/microbench:amd64-latest bench --benchmark_list_testsCopy to clipboardErrorCopied

You can run benchmarks in the following way (probably with the --benchmark_filter flag).

nvidia-docker run --privileged --rm -v `readlink -f .`:/data -u `id -u`:`id -g` raiproject/microbench:amd64-latest ./numa-separate-process.sh dgx bench /data/sync2Copy to clipboardErrorCopied

• --privileged is needed to set the NUMA policy if NUMA benchmarks are to be run.
• -v `readlink -f .`:/data maps the current directory into the container as /data.
• --benchmark_out=/data/\`hostname`.json tells the bench binary to write the json output files to /data in the container, which is mapped to the current directory.
• -u `id -u`:`id -g` tells docker to run as user id -u and group id -g, which is the current user and group. This means that files that docker produces will be modifiable from the host system without root permission.

Hunter Toolchain File

If some of the third-party code compiled by hunter needs a different compiler, you can create a cmake toolchain file to set various cmake variables that will be globally used when building that code. You can then pass this file into cmake

cmake -DCMAKE_TOOLCHAIN_FILE=toolchain.cmake ...Copy to clipboardErrorCopied

Adding a new benchmark

If you would like to develop a benchmark suite, read here for more information.
Also, check out the Example|Scope for a template to get started

Third-Party Resources

• https://github.com/google/benchmark