利用英特尔OAP和Optane技术加速大数据分析

利用英特尔OAP和Optane技术加速大数据分析2021网易数字+大会魏剑（Intel 傲腾事业部TSS经理）徐铖(Intel 人工智能与数据分析事业部研发经理)英特尔亚太研发有限公司网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会 Notices and Disclaimers2© 2018 Intel Corporation. Intel, the Intel logo, 3D XPoint, Optane, Xeon, Xeon logos, and Intel Optane logo are trademarks ofIntel Corporation in the U.S. and/or other countries. All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.The cost reduction scenarios described are intended to enable you to get a better understanding of how the purchase of a given Intel based product, combined with a number of situation-specific variables, might affect future costs and savings.Circumstances will vary and there may be unaccounted-for costs related to theuse and deployment of a given product.Nothing in this document should be interpreted as either a promise of or contract for a given level of costs or cost reduction.The benchmark results reported above may need to be revised as additional testing is conducted. The results depend on the specific platform configurations and workloads utilized in the testing, and may not be applicable to any particular user’s components, computer system or workloads. The results are not necessarily representative of other benchmarks and other benchmark results may show greater or lesser impact from mitigations.Results have been estimated based on tests conducted on pre-production systems, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance results are based on testing as of 03-14-2019 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to www.intel.com/benchmarks.Intel processors of the same SKU may vary in frequency or power as a result of natural variability in the production process.Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice Revision #20110804.Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase.For more complete information about performance and benchmarkresults, visit www.intel.com/benchmarks.*Other names and brands may be claimed as the property of others.Intel Confidential, Share Under NDA网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会 网易数字+大会议题•OAP 概述•OAP Analytic cache 介绍•Optane PMEM 介绍•OAP Analytic cache 特性介绍•Native Parquet Reader•缓存及其多种模式•算子下放(Filter/Project/Aggregation)•后续工作网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会 网易数字+大会Optimized Analytics Package (OAP) 概述OAP是Intel和相关社区维护的开源项目，旨在优化和扩展现有Spark的性能和功能。•Spark在多个方面都表现地性能卓越，但进一步提升现有的Spark性能还是存在着新的挑战：•现有的基于JVM的和基于row的计算引擎让Spark不能充分利用上Intel硬件的一些特性，比如AVX/AVX512, GPU•目前实现层面，比如内存管理，shuffle实现，都没有考虑到最新高级硬件比如Optane PMEM•批处理在很多时候还是不能满足高性能要求。https://github.com/oap-project/ 网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会 网易数字+大会Optimized Analytics Package (OAP) 组件网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会网易数字+大会