T9.02 Rapid Development of Advanced High-Speed Aerosciences Simulation Capability

These are topics we are currently preparing bids for. If you have the skill set to assist in these focus areas please submit your information through our Careers page.

• Lead Center: ARC

  Participating Center(s): JSC, LaRC

  Solicitation Year: 2021

  Scope Title:

  Aerothermal Simulation on Advanced Computer Architectures

  Scope Description:

  Aerothermodynamic simulations of planetary entry vehicles such as Orion and Dragonfly are complex and time consuming. These simulations, which solve the multispecies, multitemperature Navier-Stokes equations, require detailed models of the chemical and thermal nonequilibrium processes that take place in high-temperature shock layers. Numerical solution of these models results in a large system of highly nonlinear equations that are exceptionally stiff and difficult to solve efficiently. As a result, aerothermal simulations routinely consume 20 to 50 times the compute resources required by more conventional supersonic computational fluid dynamics (CFD) analysis, limiting the number of simulations delivered in a typical engineering design cycle to only a few dozen. Moreover, entry system designs are rapidly increasing in complexity, and unsteady flow phenomena such as supersonic retropropulsion are becoming critical considerations in their design. This increases the compute resources required for aerothermal simulation by an additional one to two orders of magnitude, which precludes the delivery of such simulations in engineering-relevant timescales.

   

  In order to deliver the aerothermal simulations required for NASA’s next generation of entry systems, access to greatly expanded compute resources is required. However, scaling up conventional central processing unit (CPU-) based supercomputers is problematic due to cost and power constraints. Many-core accelerators, such as Nvidia’s general-purpose graphical processing units (GPGPUs), offer increased compute capability with reduced cost and power requirements and are seeing rapid adoption in top-end supercomputers. As of June 2020, 144 of the top 500 fastest supercomputers leveraged accelerators or co-processors, including 6 of the top 10 [1]. All three of the U.S. Department of Energy’s upcoming exascale supercomputers will be accelerated using GPGPUs [2]. NASA has deployed Nvidia v100 GPGPUs to the Pleiades supercomputer [3]. Critically, NASA’s aerothermal simulation tools are fundamentally unable to run on many-core accelerators, and must be reengineered from the ground up to efficiently exploit such devices.

   

  This scope seeks to revolutionize NASA’s aerothermal analysis capability by developing novel simulation tools capable of efficiently targeting the advanced computational accelerators that are rapidly becoming standard in the world’s fastest supercomputers. A successful solution within this scope would demonstrate efficient simulation of a large-scale aerothermal problem of relevance on an advanced many-core architecture, for example, the Nvidia Volta GPGPU, using a prototype software package.

  Expected TRL or TRL Range at completion of the Project: 2 to 5 
  Primary Technology Taxonomy: 
  Level 1: TX 09 Entry, Descent, and Landing 
  Level 2: TX 09.1 Aeroassist and Atmospheric Entry 
  Desired Deliverables of Phase I and Phase II:

  • Software

  Desired Deliverables Description:

  The desired deliverable at the conclusion of Phase I is a prototype software package capable of solving the multispecies, multitemperature, reacting Euler equations on an advanced many-core accelerator such as an Nvidia v100 GPGPU. Parallelization across multiple accelerators and across nodes is not required. The solver shall demonstrate offloading of all primary compute kernels to the accelerator, but may do so in a nonoptimal fashion, for example, using managed memory, serializing communication and computation, etc. Some noncritical kernels such as boundary condition evaluation may still be performed on a CPU. The solver shall demonstrate kernel speedups (excluding memory transfer time) when comparing a single accelerator to a modern CPU-based, dual-socket compute node. However, overall application speedup is not expected at this stage. The solver shall be demonstrated for a relevant planetary entry vehicle such as FIRE-II, Apollo, Orion, or the Mars Science Laboratory.

   

  A successful Phase II deliverable will mature the Phase I prototype into a product ready for mission use and commercialization. Kernels for evaluating viscous fluxes shall be added, enabling computation of laminar convective heat transfer to the vehicle. Parallelization across multiple accelerators and multiple compute nodes shall be added. Good weak scaling shall be demonstrated for large 3D simulations (>10M grid cells). The implementation shall be sufficiently optimized to achieve an ~5x reduction in time-to-solution compared to NASA's Data-Parallel Line Relaxation (DPLR) aerothermal simulation code, assuming each dual-socket compute node is replaced by a single accelerator (i.e., delivered software running on eight GPGPUs shall be 5 times faster than DPLR running on eight modern, dual-socket compute nodes). Finally, the accuracy of the delivered software shall be verified by comparing to the DPLR and/or LAURA codes. The verification study shall consider flight conditions from at least two of the following planetary destinations: Earth, Mars, Titan, Venus, and Uranus/Neptune.

  State of the Art and Critical Gaps:

  NASA’s existing aerothermal analysis codes (LAURA, DPLR, US3D, etc.) all utilize domain-decomposition strategies to implement coarse-grained, distributed-memory parallelization across hundreds or thousands of conventional CPU cores. These codes are fundamentally unable to efficiently exploit many-core accelerators, which require the use of fine-grained, shared-memory parallelism over hundreds of thousands of compute elements. Addressing this gap requires reengineering our tools from the ground up and developing new algorithms that expose more parallelism and scale well to small grain sizes.

   

  Many-core accelerated CFD solvers exist in academia, industry, and government. Notable examples are PyFR from Imperial College London [4], the Ansys Fluent commercial solver [5], and NASA Langley’s FUN3D code, which recently demonstrated a 30x improvement in node-level performance using Nvidia v100 GPUs [6]. However, nearly all previous work has focused on perfect gas flow models, which have different algorithmic and resource requirements compared to real gas models. The Sandia Parallel Aerodynamics and Reentry Code (SPARC) solver is the only project of note to have demonstrated efficient real-gas capability at scale using many-core accelerators [7].

  Relevance / Science Traceability:

  This scope is directly relevant to NASA space missions in both HEOMD and SMD with an entry, descent, and landing (EDL) segment. These missions depend on aerothermal CFD to define critical flight environments and would derive large, recurring benefits from a more responsive and scalable simulation capability. This scope also has potential cross-cutting benefits for tools used by ARMD to simulate airbreathing hypersonic vehicles. Furthermore, this scope directly supports NASA’s CFD Vision 2030 Study, which calls for sustained investment to ensure that NASA’s computational aeroscience capabilities can effectively utilize the massively parallel, heterogeneous (i.e., GPU-accelerated) supercomputers expected to be the norm in 2030.

  References:

  1. “Japan Captures TOP500 Crown with Arm-Powered Supercomputer(link is external),” June 22, 2020.
  2. R. Smith: “El Capitan Supercomputer Detailed: AMD CPUs & GPUs To Drive 2 Exaflops of Compute(link is external),” March 4, 2020.
  3. NASA HECC Knowledge Base: “New NVIDIA V100 Nodes Available,” 21 June, 2019.
  4. F. Witherden, et al.: "PyFR: An Open Source Framework for Solving Advection–Diffusion Type Problems on Streaming Architectures Using the Flux Reconstruction Approach(link is external)," Computer Physics Comm., vol. 185, no. 11, pp. 3028-3040, 2014.
  5. V. Sellappan and B. Desam: "Accelerating ANSYS Fluent Simulations with NVIDIA GPUs(link is external)," 2015.
  6. E. Neilsen, et al.: "Unstructured Grid CFD Algorithms for NVIDIA GPUs(link is external)," March 2019.
  7. M. Howard, et al.: "Employing Multiple Levels of Parallelism for CFD at Large Scales on Next Generation High-Performance Computing Platforms(link is external)," ICCFD10, Barcelona, Spain, 2018.
  8. J. Slotnick, et al.: “CFD Vision 2030 Study: A Path to Revolutionary Computational Aerosciences,” NASA/CR–2014-218178, March 2014.