MPI, GPU, and multi-GPU execution
Gaugefields v1 uses the LatticeMatrices/JACC backend for distributed fields. There is no separate MPI constructor and no accelerator flag in the v1 API. The simulation source is selected by process_grid; JACC determines where its arrays and kernels run.
MPI.jl is optional. Ordinary serial CPU and single-GPU programs do not need it. MPI applications must load MPI before constructing an MPI-backed field. Gaugefields calls MPI.Init() lazily on first MPI use and never calls MPI.Finalize(). Call MPI.Init(...) explicitly first when custom initialization options such as the thread level are needed.
Complete MPI example
The application environment must list MPI and JACC directly:
pkg> add Gaugefields JACC MPISave the following as mpi_example.jl:
using MPI
import JACC
JACC.@init_backend
using Gaugefields
MPI.Init()
world = MPI.COMM_WORLD
nranks = MPI.Comm_size(world)
lattice = (8, 8, 8, 8 * nranks)
process_grid = (1, 1, 1, nranks)
U = gauge_configuration(
lattice;
colors=3,
halo=1,
start=:hot,
seed=0x1234,
process_grid=process_grid,
comm=world,
)
comm = gauge_communicator(U)
rank = MPI.Comm_rank(comm)
plaq = measure_plaquette(U)
rank == 0 && println("plaquette = ", plaq)
flow = gradient_flow(U; steps=2, step_size=0.01)
flow!(U, flow)
rank == 0 && println("flowed plaquette = ", measure_plaquette(U))
MPI.Barrier(comm)
MPI.Finalize()The process grid has one entry per lattice direction. It must satisfy:
prod(process_grid) == nranks;- every global extent is divisible by the corresponding grid entry;
- local extents must be large enough for the selected halo and operation.
When process_grid is omitted or set to :auto, Gaugefields chooses a valid factorization of the communicator size that respects lattice divisibility and minimizes a surface-to-volume score. Pass an explicit tuple when a particular topology is required. comm may be MPI.COMM_SELF, MPI.COMM_WORLD, or a subcommunicator; Gaugefields does not call MPI.Finalize().
Even with one MPI rank, the communicator choice remains explicit. Omitting comm after using MPI uses MPI.COMM_WORLD; pass comm=SerialCommunicator() to use the serial communication path instead.
CPU execution
Select the threads backend once, then restart Julia:
import JACC
JACC.set_backend("threads")Launch MPI with the desired threads per rank:
mpiexec -n 4 julia --threads=4 --project=. mpi_example.jlOne GPU
Select the backend once and restart Julia:
import JACC
JACC.set_backend("cuda") # NVIDIA
# JACC.set_backend("amdgpu") # AMD
# JACC.set_backend("oneapi") # IntelRun the same Julia source with one process. No Gaugefields GPU keyword is needed.
Multiple GPUs
Launch one MPI rank per GPU with the same mpi_example.jl:
mpiexec -n 4 julia --project=. mpi_example.jlLatticeMatrices maps node-local ranks to visible devices. The number of ranks on a node must not exceed the GPUs visible to those ranks. Scheduler resource flags are system-specific, but process_grid remains the only decomposition setting in Gaugefields code.
Direct device-buffer communication requires an MPI installation built for the selected GPU runtime. In particular, CUDA multi-GPU runs require CUDA-aware MPI, and MPI.jl's libmpi must match the mpiexec used to launch the job.
Portable algorithms
The following v1 interfaces use the same LM configuration on CPU, GPU, MPI, and multi-GPU:
measure_plaquetteandmeasure_polyakov_loop;gradient_flowandflow!;heatbath_updater,heatbath!, andoverrelaxation!;stout_smearingandsmear;gaussian_momenta,gaussian_momenta!,md_driver, andmd_trajectory!;save_configurationand in-placeload_configuration!where supported by the selected file format.
All ranks must execute collective algorithms in the same order.
JLD2 is the portable checkpoint format for serial, MPI, GPU, and multi-GPU fields. Saving is collective: rank 0 reconstructs the global physical links on host memory and writes one file. Loading reconstructs the destination on its current devices and process grid. All ranks must call save and load in the same order.
NERSC SU(3) I/O also supports CPU, GPU, MPI, and multi-GPU fields through format=:nersc, including allocating load_configuration. It uses collective host staging on rank zero and converts file data to the destination precision before device transfer. All ranks participate; see the NERSC I/O guide for examples and format options.
Reproducibility across decompositions
An explicit seed uses global-site streams for LM hot starts, Gaussian momenta, heatbath, and overrelaxation. Keeping the seed, algorithm, and logical sweep counter fixed makes these streams independent of the MPI process-grid decomposition.
A complete HMC chain must additionally synchronize or broadcast its Metropolis decision. See Randomness and reproducibility and HMC and custom integrators.
The pre-v1 mpi, PEs, isMPILattice, cuda, and accelerator constructor flags are documented only in Legacy API.