MPI, GPU, and multi-GPU execution
Gaugefields v1 uses the LatticeMatrices/JACC backend for distributed fields. There is no separate MPI constructor and no accelerator flag in the v1 API. The simulation source is selected by process_grid; JACC determines where its arrays and kernels run.
Complete MPI example
The application environment must list MPI and JACC directly:
pkg> add Gaugefields JACC MPISave the following as mpi_example.jl:
using MPI
import JACC
JACC.@init_backend
using Gaugefields
MPI.Init()
world = MPI.COMM_WORLD
nranks = MPI.Comm_size(world)
lattice = (8, 8, 8, 8 * nranks)
process_grid = (1, 1, 1, nranks)
U = gauge_configuration(
lattice;
colors=3,
halo=1,
start=:hot,
seed=0x1234,
process_grid=process_grid,
)
comm = gauge_communicator(U)
rank = MPI.Comm_rank(comm)
plaq = measure_plaquette(U)
rank == 0 && println("plaquette = ", plaq)
flow = gradient_flow(U; steps=2, step_size=0.01)
flow!(U, flow)
rank == 0 && println("flowed plaquette = ", measure_plaquette(U))
MPI.Barrier(comm)
MPI.Finalize()The process grid has one entry per lattice direction. It must satisfy:
prod(process_grid) == nranks;- every global extent is divisible by the corresponding grid entry;
- local extents must be large enough for the selected halo and operation.
When process_grid is omitted in 4D, the default decomposition places all ranks in the final direction.
CPU execution
Select the threads backend once, then restart Julia:
import JACC
JACC.set_backend("threads")Launch MPI with the desired threads per rank:
mpiexec -n 4 julia --threads=4 --project=. mpi_example.jlOne GPU
Select the backend once and restart Julia:
import JACC
JACC.set_backend("cuda") # NVIDIA
# JACC.set_backend("amdgpu") # AMD
# JACC.set_backend("oneapi") # IntelRun the same Julia source with one process. No Gaugefields GPU keyword is needed.
Multiple GPUs
Launch one MPI rank per GPU with the same mpi_example.jl:
mpiexec -n 4 julia --project=. mpi_example.jlLatticeMatrices maps node-local ranks to visible devices. The number of ranks on a node must not exceed the GPUs visible to those ranks. Scheduler resource flags are system-specific, but process_grid remains the only decomposition setting in Gaugefields code.
Direct device-buffer communication requires an MPI installation built for the selected GPU runtime. In particular, CUDA multi-GPU runs require CUDA-aware MPI, and MPI.jl's libmpi must match the mpiexec used to launch the job.
Portable algorithms
The following v1 interfaces use the same LM configuration on CPU, GPU, MPI, and multi-GPU:
measure_plaquetteandmeasure_polyakov_loop;gradient_flowandflow!;heatbath_updater,heatbath!, andoverrelaxation!;stout_smearingandsmear;gaussian_momenta,md_driver, andmd_trajectory!;save_configurationand in-placeload_configuration!where supported by the selected file format.
All ranks must execute collective algorithms in the same order.
Reproducibility across decompositions
An explicit seed uses global-site streams for LM hot starts, Gaussian momenta, heatbath, and overrelaxation. Keeping the seed, algorithm, and logical sweep counter fixed makes these streams independent of the MPI process-grid decomposition.
A complete HMC chain must additionally synchronize or broadcast its Metropolis decision. See Randomness and reproducibility and HMC and custom integrators.
The pre-v1 mpi, PEs, isMPILattice, cuda, and accelerator constructor flags are documented only in Legacy API.