OAP Developer Guide

This document contains the instructions & scripts on installing necessary dependencies and building OAP modules. You can get more detailed information from OAP each module below.

Building OAP

Prerequisites

We provide scripts to help automatically install dependencies required, please change to root user and run:

# git clone -b <tag-version> https://github.com/oap-project/oap-tools.git
# cd oap-tools
# sh dev/install-compile-time-dependencies.sh

Note: oap-tools tag version v1.2.0 corresponds to all OAP modules' tag version v1.2.0.

Then the dependencies below will be installed:

Cmake
GCC > 7
Memkind
Vmemcache
HPNL
PMDK
OneAPI
Arrow
LLVM
Requirements for Shuffle Remote PMem Extension
If enable Shuffle Remote PMem extension with RDMA, you can refer to PMem Shuffle to configure and validate RDMA in advance.

Building

Building OAP

OAP is built with Apache Maven and Oracle Java 8.

To build OAP package, run command below then you can find a tarball named oap-$VERSION-*.tar.gz under directory $OAP_TOOLS_HOME/dev/release-package, which contains all OAP module jars. Change to root user, run

# cd oap-tools
# sh dev/compile-oap.sh

Building OAP specific module

If you just want to build a specific OAP Module, such as sql-ds-cache, change to root user, then run:

# cd oap-tools
# sh dev/compile-oap.sh --component=sql-ds-cache

Enabling NUMA binding for PMem in Spark

Rebuilding Spark packages with NUMA binding patch

When using PMem as a cache medium apply the NUMA binding patch numa-binding-spark-3.1.1.patch to Spark source code for best performance.

Download src for Spark-3.1.1 and clone the src from github.
Apply this patch and rebuild the Spark package.

git apply numa-binding-spark-3.1.1.patch

Add these configuration items to the Spark configuration file $SPARK_HOME/conf/spark-defaults.conf to enable NUMA binding.

spark.yarn.numa.enabled true