Reduce the memory usage that is important for ne1024 simulation

[sjsprecious]

This PR reduces the memory usage that is critical for ultra-high resolution simulation such as ne1024. All the code changes are done by Claude under my supervisory.

The goal is to let the lnd→rof conservative coupling maps be read from offline weight files instead of being computed online, which OOM-kills the job at ne1024 during DataInitialize (ESMF_FieldRegridStore → GeomRendezvous → Zoltan_RCB). This is the implementation of the lnd2rof_consf OOM fix. There are two distinct lnd→rof maps handled, plus an aoflux memory optimization.

1. Flux/runoff coupling map — bug fix

In main, the lnd2rof_map attribute (driven by the pre-existing LND2ROF_FMAPNAME XML var) was already read into the lnd2rof_map variable — but the five addmap_from calls for the runoff fields (Flrl_rofsur, Flrl_rofi, Flrl_rofgwl, Flrl_rofsub, Flrl_irrig) hardcoded unset, so the file was silently ignored and the map was always built online.

2. Fraction-init map — new feature

This is a separate map (destarea, not fracarea) used during med_fraction_init, with no pre-existing namelist hook. The fix adds full new plumbing:

• New XML entry LND2ROF_FRAC_FMAPNAME (default unset, env_run.xml, group run_domain) in config_component.xml.
• New driver namelist lnd2rof_fmap (modify_via_xml="LND2ROF_FRAC_FMAPNAME") in namelist_definition_drv.xml.
• med_fraction_mod.F90: reads the lnd2rof_fmap attribute via NUOPC_CompAttributeGet; if it is present, set, and the file exists on disk (inquire), it calls med_map_routehandles_init with mapfile= to read offline weights; otherwise it falls back to the original online path — no behavior change when unset. Adds a use NUOPC import and local vars (isPresent/isSet/lexist, lnd2rof_fmap).
• med_map_mod.F90: threads a new optional, intent(in) :: mapfile argument through med_map_routehandles_initfrom_fieldbundle, forwarding it to the field-level routine (which already accepted mapfile) when present. Backward-compatible.

3. aoflux mesh memory optimization

Replaces is_local%wrap%aoflux_mesh = ESMF_MeshCreate(lmesh, rc=rc) with is_local%wrap%aoflux_mesh = lmesh — reuses the ocean field's existing mesh handle instead of allocating a duplicate full ESMF mesh per rank at ne1024pg2. Safe because lmesh persists in FBArea(compocn) and aoflux_mesh is never destroyed.

[billsacks]

@mvertens - If you have a chance, I'd like to hear any thoughts you have on this change. (I have just skimmed it so far - haven't had a chance to look closely yet.)

[billsacks]

The overall changes here look good to me. I appreciate your plugging in the use of lnd2rof_map, which seems like it previously wasn't hooked up. And I think I see why you needed to introduce a separate lnd2rof_fmap (though see my comments asking for this to be made more explicit in some documentation).

I do have a few requests... many about editing some comments, but a couple slightly more substantial... but still overall minor - for the most part this looks good to me.

Once you make the final changes, I'd like to see that at least one or two tests have been run with baseline comparisons to verify that these changes work and are bit-for-bit in out-of-the-box configurations. One test that I think would cover all of these changes would be a B compset test with the aoflux grid set to ogrid (the setting of ogrid is needed to cover the changes in med_phases_aofluxes_mod, and should still cover the other changes here). I'd like to see that run with comparisons against a baseline.

[jtruesdal]

@sjsprecious and @billsacks I was also thinking about a test to show that the offline and inline are identical but in the past I've run into roundoff errors in the way the model calculates the mesh compared to offline. If that's the case here you may want to implement some type of tolerance. Maybe a nonissue if they are identical in this case. Other than that I think with the current set of updates the code looks good to me.

[sjsprecious]

Thanks @jtruesdal for your comments. I think one thing that can contribute to the difference here is that users may use different compiler/ESMF versions when generating the offline mesh.

Once Bill provides the instructions about running the tests he suggested, I will let you know at least whether the code changes affect the current behavior.

[billsacks]

The latest set of changes looks good - thank you @sjsprecious for making those changes.

A piece I am less confident about is whether the assignment of aoflux_mesh to lmesh (as opposed to doing an ESMF_MeshCreate there) is always safe. I am asking the ESMF group about that, along with the similar change you made in CDEPS.

Regarding testing, a good test would be SMS_D_Ld1.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-aoflux_ogrid, since this covers the change to med_phases_aofluxes_mod in addition to your other changes. This is in the prebeta test list. It hasn't been run on recent versions of the code, so you'll need to generate baselines and then run with your latest changes with comparisons against baselines.

It would also be good to run aux_cime_baselines, or at least keep a careful eye on aux_cime_baselines when they are run in their nightly run following the merge of this PR. As with the associated CDEPS PR, @fischer-ncar could provide some input here on testing.

Regarding @jtruesdal 's point: I wasn't thinking of something as rigorous as testing behavior with online vs. offline generation of mappings - I'm mainly wanting to confirm that the changes here don't break anything or change answers for the case where we're still using online mapping.

[billsacks]

I talked with @mvertens today.

The good news is that she feels that this is NOT a problem:

A piece I am less confident about is whether the assignment of aoflux_mesh to lmesh (as opposed to doing an ESMF_MeshCreate there) is always safe. I am asking the ESMF group about that, along with the similar change you made in CDEPS.

However, @mvertens raised a question / concern about having two different mapping files: LND2ROF_FRAC_FMAPNAME and LND2ROF_FMAPNAME. I think I see why this was done, but @mvertens questioned whether this was necessary.

Our initial question on this is: What is the difference in how these mapping files are generated? My sense is that LND2ROF_FMAPNAME should be generated with --norm_type fracarea whereas LND2ROF_FRAC_FMAPNAME should be generated with --norm_type dstarea (or no --norm_type, since dstarea is the default). Is that what was done? If so, can you add some documentation of this difference to the relevant xml and namelist definition documentation? We are also wondering if the resulting mapping files actually differ based on using these different --norm_type flags. Can you comment on this?

This discussion led me to find these related issues:

• Change mapconsf mappings to mapconsd? #408
• Have temporary ability to set lfrac mapping for mizuRoute from mapconsd to mapconsf #409
• CTSM runoff remapping issue again mizuRoute#426

The summary of those issues is: Even though we currently have different mapping types for different lnd2rof mappings - and that's what you're supporting in the changes in this PR - this is considered to be a bug rather than a feature, and in fact we want to be using the same mapping type for all the mappings. So now I'm not sure what the right path forward is. I can see a few paths forward, which I lay out below... this probably needs some more discussion to figure out how we want to proceed.

(1) Keep the current changes and merge this as is

Pros:

• Allows high-res runs to be done from main branch
• Specifying pre-made mapping files will give the same behavior as you currently get

Cons:

• Confusing for people using this, since the differences between the two LND2ROF maps are subtle, and extra work is needed to create both maps - and a user needs to know that they need to create both maps)
• This will probably be reworked in the future, so we'll need to rework the changes here at that point

(2) Make an initial change where we change all of the lnd2rof mapping to be consd instead of consf, then change this PR to use a single LND2ROF mapping file and merge it

Pros:

• Allows high-res runs to be done from main branch
• Specifying pre-made mapping files will give the same behavior as online mapping (though note that this behavior will differ from the current behavior)
• Simplifies the LND2ROF mapping files
• Moves us in the direction we want to go, according to Change mapconsf mappings to mapconsd? #408

Cons:

• I'd like to have more discussion of whether it's really correct to change all of the consf mappings to be consd as suggested in Change mapconsf mappings to mapconsd? #408... so this will take more time
• The initial step may be answer changing; would want to investigate by how much
• If anyone has lnd2rof mapping files already, and if these were generated with --norm_type fracarea, they will need to be regenerated with --norm_type dstarea

(3) Don't change online mapping behavior for now, but use the same mapping file for the lnd2rof frac mapping as for the other lnd2rof mappings... this relies on the assumption that the differences between consf and consd mapping are inconsequential (which #408 suggests, but I'm not sure it applies in all cases). Bring this to main in this way.

Pros:

• Allows high-res runs to be done from main branch
• Simplifies the LND2ROF mapping files
• Moves us forward in the right direction, though not as far as option (2)

Cons:

• Answers could differ between using a pre-computed mapping file vs. online mapping; this may be doing (slightly) the wrong thing for some of the lnd2rof mappings when using a pre-computed mapping file. We'd want to verify that the differences are small.
• Puts some pressure on to resolve Change mapconsf mappings to mapconsd? #408 (at least for the lnd2rof mappings) to remove the above con.

(4) Defer bringing in these changes until post CESM3

Pros:

• Avoids both (a) changing CESM results for now and (b) making changes that might be confusing or need to be redone later.
• Specifying pre-made mapping files will give the same behavior as you currently get

Cons:

• High-res runs will need to be done using this branch, which may need to be maintained for some period

(5) Hybrid between (1) and (4): bring in the changes on this branch that do NOT rely on the new mapping file (LND2ROF_FRAC_FMAPNAME); leave those additional changes on a branch for now.

[sjsprecious]

Thanks @billsacks .

Our initial question on this is: What is the difference in how these mapping files are generated? My sense is that LND2ROF_FMAPNAME should be generated with --norm_type fracarea whereas LND2ROF_FRAC_FMAPNAME should be generated with --norm_type dstarea (or no --norm_type, since dstarea is the default). Is that what was done? If so, can you add some documentation of this difference to the relevant xml and namelist definition documentation?

Yes, LND2ROF_FMAPNAME is generated with --norm_type fracarea whereas LND2ROF_FRAC_FMAPNAME is generated with --norm_type dstarea. I have updated the comments in the XML files as suggested.

We are also wondering if the resulting mapping files actually differ based on using these different --norm_type flags. Can you comment on this?

Most fields between these two mapping files using different --norm_type options are identical. But the weights variable S does differ around 1.45e-13 (maximum absolute difference) or 6.7e-13 (maximum relative difference). Thus I would say they are not exactly the same but the difference seems subtle or at roundoff level.

So now I'm not sure what the right path forward is. I can see a few paths forward, which I lay out below... this probably needs some more discussion to figure out how we want to proceed.

I appreciate that you list all the available options here and provide detailed explanations. My two cents is that if my changes pass the regression tests that you suggested, it means that they do not change the current CMEPS behavior and should be safe to bring into CESM. I understand that providing two similar mapping files may be confusing, but if those XML variables are unset, the online mapping is used and the user may not even notice the existence of those XML variables.

Regarding the option to use the same consd or lnd2rof mapping, I feel that it is out of the scope of this PR and should be addressed by a separate PR (you have also listed a few related open issues), especially that those revisions may cause answer changing while this PR is expected to be BFB. I think it is fine to revise some work from PR afterwards once you decide how to address those issues.