NOTE - I don't really understand the problem here. I was using a bot to debug something else (MOAB/MBDA) and it stumbled on this. I asked the bot to write up the issue below, so feel free to close this if it doesn't make sense. I have personally confirmed the reproducer, the crash, and the nc_driver=mpiio workaround on the OLCF Andes machine.
Summary
PnetCDF 1.15.0 crashes with SIGFPE (integer divide-by-zero) inside GIOI_Lustre_open() when
opening any existing file on a Lustre filesystem, if the library was built without Lustre
API support (HAVE_LUSTRE undefined). The GIO layer selects the Lustre driver at runtime based
on the statfs() f_type magic (gio/src/gio_fstype.c), regardless of whether the build has any
way to query file striping — and the open path then divides two zero-initialized values.
The crash occurs on the very first ncmpi_open() call, with a single process, for any file
format (CDF-1/2/5) and any striping layout (reproduced with both plain stripe_count=1 layouts
and PFL/DoM composite layouts).
This affects the current conda-forge libpnetcdf 1.15.0 packages (built without liblustreapi
— the shared library contains no llapi_* references), so any conda-forge-based application
(e.g. MOAB tools) crashes out of the box on Lustre systems.
Environment
- PnetCDF 1.15.0 ("PnetCDF library version 1.15.0 of July 1, 2026"), conda-forge build
libpnetcdf 1.15.0 mpi_mpich_hb983576_201
- MPICH 5.0.1 (conda-forge), x86_64
- OLCF Andes (login node), RHEL 8.10, kernel 4.18.0-553
- Lustre client 2.15.3.2_cray (Orion/
/lustre/orion filesystem)
Reproducer
#include <stdio.h>
#include <mpi.h>
#include <pnetcdf.h>
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
int ncid, err;
err = ncmpi_open(MPI_COMM_WORLD, argv[1], NC_NOWRITE, MPI_INFO_NULL, &ncid);
printf("ncmpi_open: err=%d (%s)\n", err, ncmpi_strerror(err));
MPI_Finalize();
return 0;
}
$ mpicc pnc_test.c -o pnc_test -lpnetcdf
$ ./pnc_test /lustre/orion/.../any_valid_file.nc # single process, no mpiexec needed
Floating point exception (core dumped)
The same binary run on a non-Lustre filesystem (or with PNETCDF_HINTS="nc_driver=mpiio")
works fine, confirming the file itself is valid.
Backtrace
Program terminated with signal SIGFPE, Arithmetic exception.
#0 0x... in GIOI_Lustre_open () from libpnetcdf.so.8
#1 0x... in GIO_open ()
#2 0x... in ncmpio_open ()
#3 0x... in ncmpi_open ()
#4 0x... in main ()
=> GIOI_Lustre_open+245: idiv %esi
rax 0x0 0
rsi 0x0 0 <- 0 / 0
Root cause
In gio/src/gio_lustre_open.c, GIOI_Lustre_open() (line 1173 in the 1.15.0 release tarball):
stripe_count, numOSTs are initialized to 0 (lines ~1188-1191).
- The code that queries actual striping is guarded by
#ifdef HAVE_LUSTRE (llapi path) /
#elif defined(MIMIC_LUSTRE). When neither is defined at build time, both variables
remain 0.
- After the
MPI_Bcast of stripin_info[], the open path only checks
stripin_info[0] != GIO_NOERR before computing (line 1251):
fh->hints->overstriping_ratio = stripin_info[2] / stripin_info[4]; /* 0 / 0 -> SIGFPE */
Note the create path has the missing guard (line ~1132 checks stripin_info[4] == 0 before
the equivalent division at line 1142), but the open path does not.
Additional latent issues even for builds with HAVE_LUSTRE:
get_striping() explicitly return 0 when llapi_layout_stripe_count_get() reports
LLAPI_LAYOUT_DEFAULT / LLAPI_LAYOUT_WIDE / invalid (lines ~380-386). On systems where the
filesystem default is a PFL/composite layout (e.g. OLCF Orion, which uses progressive layouts
with a Data-on-MDT first component reporting stripe_count: 0), this returns 0 for ordinary
files created with default striping, which would feed the same unguarded division in the open
path.
Lustre_set_cb_node_list() divides by striping_factor
(rounds = num_aggr / striping_factor, line 689), which is also unguarded if
striping_factor == 0 gets that far.
GIOI_Lustre_open_on_demand() may warrant the same check.
Suggested fix
- In
GIOI_Lustre_open(), mirror the create path's guard: check stripin_info[4] == 0 (and
stripin_info[2] <= 0) after the Bcast, and either fall back to sane defaults (e.g.
striping_factor=1, overstriping_ratio=1) or fall back to the generic/MPI-IO path instead
of dividing.
- Consider not selecting the Lustre GIO driver at all when the library is built without
HAVE_LUSTRE/MIMIC_LUSTRE, since it has no way to obtain striping information.
Workaround
PNETCDF_HINTS="nc_driver=mpiio" avoids the GIO layer entirely and works correctly.
NOTE - I don't really understand the problem here. I was using a bot to debug something else (MOAB/MBDA) and it stumbled on this. I asked the bot to write up the issue below, so feel free to close this if it doesn't make sense. I have personally confirmed the reproducer, the crash, and the
nc_driver=mpiioworkaround on the OLCF Andes machine.Summary
PnetCDF 1.15.0 crashes with SIGFPE (integer divide-by-zero) inside
GIOI_Lustre_open()whenopening any existing file on a Lustre filesystem, if the library was built without Lustre
API support (
HAVE_LUSTREundefined). The GIO layer selects the Lustre driver at runtime basedon the
statfs()f_type magic (gio/src/gio_fstype.c), regardless of whether the build has anyway to query file striping — and the open path then divides two zero-initialized values.
The crash occurs on the very first
ncmpi_open()call, with a single process, for any fileformat (CDF-1/2/5) and any striping layout (reproduced with both plain
stripe_count=1layoutsand PFL/DoM composite layouts).
This affects the current conda-forge
libpnetcdf 1.15.0packages (built withoutliblustreapi— the shared library contains no
llapi_*references), so any conda-forge-based application(e.g. MOAB tools) crashes out of the box on Lustre systems.
Environment
libpnetcdf 1.15.0 mpi_mpich_hb983576_201/lustre/orionfilesystem)Reproducer
The same binary run on a non-Lustre filesystem (or with
PNETCDF_HINTS="nc_driver=mpiio")works fine, confirming the file itself is valid.
Backtrace
Root cause
In
gio/src/gio_lustre_open.c,GIOI_Lustre_open()(line 1173 in the 1.15.0 release tarball):stripe_count,numOSTsare initialized to 0 (lines ~1188-1191).#ifdef HAVE_LUSTRE(llapi path) /#elif defined(MIMIC_LUSTRE). When neither is defined at build time, both variablesremain 0.
MPI_Bcastofstripin_info[], the open path only checksstripin_info[0] != GIO_NOERRbefore computing (line 1251):Note the create path has the missing guard (line ~1132 checks
stripin_info[4] == 0beforethe equivalent division at line 1142), but the open path does not.
Additional latent issues even for builds with
HAVE_LUSTRE:get_striping()explicitlyreturn 0whenllapi_layout_stripe_count_get()reportsLLAPI_LAYOUT_DEFAULT/LLAPI_LAYOUT_WIDE/ invalid (lines ~380-386). On systems where thefilesystem default is a PFL/composite layout (e.g. OLCF Orion, which uses progressive layouts
with a Data-on-MDT first component reporting
stripe_count: 0), this returns 0 for ordinaryfiles created with default striping, which would feed the same unguarded division in the open
path.
Lustre_set_cb_node_list()divides bystriping_factor(
rounds = num_aggr / striping_factor, line 689), which is also unguarded ifstriping_factor == 0gets that far.GIOI_Lustre_open_on_demand()may warrant the same check.Suggested fix
GIOI_Lustre_open(), mirror the create path's guard: checkstripin_info[4] == 0(andstripin_info[2] <= 0) after the Bcast, and either fall back to sane defaults (e.g.striping_factor=1,overstriping_ratio=1) or fall back to the generic/MPI-IO path insteadof dividing.
HAVE_LUSTRE/MIMIC_LUSTRE, since it has no way to obtain striping information.Workaround
PNETCDF_HINTS="nc_driver=mpiio"avoids the GIO layer entirely and works correctly.