Abstract

High Performance Computing (HPC) technologies and solutions provide an increasingly important means for organizations and institutions to process larger volumes of data and to generate insights. In particular, big data environments, which are characterized by large and complex datasets, with an unbounded potential for data growth and a need to process both structured and unstructured data quickly, require advanced HPC solutions and technologies. HPC solutions are used to perform complex data transformations, analytics, and simulations, including solving complex numerical problems and modeling complex phenomena. The growing capabilities of HPC technologies provide an array of potential solutions for big data challenges. These include cloud computing, distributed computing, virtualization, high-performance storage, and faster networking solutions, to name a few. Cloud solutions are used for rapid provisioning and scalability of compute and storage resources, while virtualization technologies enable the runtime isolation of application components to scale applications to massive datasets. High-speed networking technologies enable better collaboration, data exchange, and data transfer within big data platforms. Distributed computing solutions, such as Apache Hadoop and Apache Spark, provide solutions for performing Map Reduce operations across clusters of commodity hardware. High-performance storage solutions, such as Alluxio, provide an efficient way to handle massive data sets, by providing a unified storage tier across multiple platforms, including in-memory, distributed file system, and object storage.