Topology-aware optimizations for multi-GPU ptychographic image reconstruction

Xiaodong Yu, Tekin Biçer, Rajkumar Kettimuthu, Ian T. Foster

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

11 Scopus citations

Abstract

Ptychography is an advanced high-resolution X-ray imaging technique that can generate extremely large datasets. Ptychographic reconstruction transforms reciprocal space experimental data to high-resolution 2D real-space images. GPUs have been used extensively to meet the computational requirements of the reconstruction. Generic multi-GPU reconstruction solutions use common communication topologies, such as P2P graph and ring, that are provided by MPI and NCCL libraries, to establish inter-GPU communications. However, these common topologies assume homogeneous physical links between GPUs, resulting in sub-optimal performance on heterogeneous configurations that are composed of both high- (e.g., NVLink) and low-speed (e.g., PCIe) interconnects. This mismatch between application-level communication topology and physical interconnection can cause data transfer congestion, inefficient memory access, and under-utilization of network resources. Here we present topology-aware designs and optimizations to address the aforementioned mismatch and boost end-to-end application performance. We introduce topology-aware data splitting, propose a novel communication topology, and incorporate asynchronous data movement and computation. We evaluate our design and optimizations using real and artificial datasets and compare its performance with that of the direct P2P and NCCL-based approaches. The results show that our optimizations always outperform the counterparts and achieve up to 5.13× and 1.63× communication and end-to-end application speedups, respectively.

Original languageEnglish
Title of host publicationICS 2021 - Proceedings of the 2021 ACM International Conference on Supercomputing
Pages354-366
Number of pages13
ISBN (Electronic)9781450383356
DOIs
StatePublished - 3 Jun 2021
Event35th ACM International Conference on Supercomputing, ICS 2021 - Virtual, Online, United States
Duration: 14 Jun 202117 Jun 2021

Publication series

NameProceedings of the International Conference on Supercomputing

Conference

Conference35th ACM International Conference on Supercomputing, ICS 2021
Country/TerritoryUnited States
CityVirtual, Online
Period14/06/2117/06/21

Keywords

  • GPU
  • Heterogeneous inter-GPU connections
  • Image reconstruction
  • NVLink
  • Neighborhood communication
  • Ptychography

Fingerprint

Dive into the research topics of 'Topology-aware optimizations for multi-GPU ptychographic image reconstruction'. Together they form a unique fingerprint.

Cite this