Overview

---

Optimized Analytics Package (OAP) is an open source project to optimize Apache Spark on Cache, Shuffle, Native SQL Engine, MLlib and so on, driven by Intel and the community.

Why use OAP?

Apache Spark is powerful and well optimized on many aspects, but still faces some challenges to achieve the higher-level performance.

• The JVM and row-based computing engine prevents Spark to be fully optimized for Intel hardware, for example AVX/AVX512, GPU

• The current implementation of key aspects, such as memory management & shuffle, doesn't consider the latest technology advancements, like PMEM

• The batch processing engine cannot satisfy the need of queries with high performance requirement.

OAP Project is targeted to optimize Spark on these aspects above, now it has 7 components, including Gazelle Plugin, SQL DS Cache, OAP MLlib, PMem Spill, PMem Common, PMem Shuffle and Remote Shuffle.

How to use OAP?

Guide

Please refer to the total OAP project installation and developer guide below.

• OAP Installation Guide
• OAP Developer Guide

Components

You can get more detailed information from each module web page of OAP Project below.

• Gazelle Plugin
• SQL DS Cache
• OAP MLlib
• PMem Shuffle
• Remote Shuffle
• PMem Spill
• PMem Common