Overview

---

Optimized Analytics Package (OAP) is an open source project to optimize Apache Spark on cache, shuffle, SQL engine, MLlib and so on, driven by Intel and the community.

Why use OAP?

Apache Spark is powerful and well optimized on many aspects, but still faces some challenges to achieve the higher-level performance.

• The JVM and row-based computing engine prevents Spark to be fully optimized for Intel hardware, for example AVX/AVX512, GPU

• The current implementation of key aspects, such as memory management & shuffle, doesn't consider the latest technology advancements, like PMEM

• The batch processing engine cannot satisfy the need of queries with high performance requirement.

OAP Project is targeted to optimize Spark on these aspects above, it had 6 components, including Gazelle Plugin, SQL DS Cache, OAP MLlib, PMem Spill, PMem Common, and PMem Shuffle in previous releases.

Currently, from OAP 1.3.1, it has 2 components including Gazelle Plugin and OAP MLlib.

How to use OAP?

Guide

Please refer to the total OAP project installation and developer guide below.

• OAP Installation Guide
• OAP Developer Guide

Components

You can get more detailed information from each module web page of OAP Project below.

• Gazelle Plugin
• OAP MLlib